-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pool: check prereqs on task spawn #6143
Comments
Good point. It would be sufficient to check the DB only when the workflow stalls, but with significant cons:
|
Yes, this would be highly confusing. I think the only way around this without checking prereqs for each spawn (not that I'm suggesting it per-se) would be to load in the task history from the DB and spawn on all outputs. If tasks already ran, nothing would happen, if they haven't then they will be spawned allowing the workflow to continue without the risk of artificial stall. However, this would have the caveat that we would be spawning according to the current graph which may differ from the graph historical tasks were running according to (i.e. it could result in spawning errors). |
When we spawn a task, we currently check the DB to retrieve previous submissions. This is required for:
Whilst we do check for previous submissions, we do not presently check task prerequisites. In some situations, task prerequisites which are satisfied in the DB, are left unsatisfied in the task pool when tasks are spawned. This can happen when tasks are added to the pool via means other than natural task satisfaction (e.g. #5952).
Easy way to replicate this:
Sadly, I don't this it is sufficient to check the DB only when such interventions are performed, because tasks downstream of the ones being added which are yet to be spawned may also be missing this state and end up with (erroneously) partially satisfied prerequisites.
This is a bad bug as it makes it look to the user like tasks which have run, haven't. It's hard to explain, especially as multiple Cylc interfaces will provide erroneous information. Moreover it's very hard to recover from as the consequences may last as long as the longest inter-cycle dependency in the workflow.
There are efficiency concerns over requesting task prerequisites, however, we haven't ascertained that this would be a problem as yet, however, DB processing is definitely a bottleneck. One way to reduce these overheads might be to only request satisfied prerequisites from the DB. Merging this request in with the existing DB request would also improve performance. As would batching the requests where multiple tasks are spawned in the same main loop iteration.
This issue has relevance to workflow extension use cases (#5952), graph changes either by reload or restart
The text was updated successfully, but these errors were encountered: