Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sub-optimal spawning of tasks that lose their parents #3758

Open
hjoliver opened this issue Aug 6, 2020 · 0 comments
Open

sub-optimal spawning of tasks that lose their parents #3758

hjoliver opened this issue Aug 6, 2020 · 0 comments
Assignees
Milestone

Comments

@hjoliver
Copy link
Member

hjoliver commented Aug 6, 2020

Parent-less tasks have no parents to spawn them "on demand", so we have to auto-spawn them:

P1 = "foo"  # foo must be auto-spawned

To stick within the runahead limit and avoid an infinte-spawning apocalypse, we do this by spawning a parent-less task when its previous instance drops out of the runahead pool into the main pool. (Above, when foo.3 leaves the runahead pool, spawn foo.4 into the runahead pool, and so on).

Note this does not impose any artificial sequential behavior on the execution of foo - so that's all fine. However, things are less optimal if foo initially has parents and then loses them:

R2/^/P1 = "dad => foo"
P1 = "foo"

Ideally (constrained only by the runahead limit) foo.3 should be able to start concurrently with dad.1 here, to run before foo.1 and foo.2 which have to wait on their parents. But with the implementation described above, foo.3 won't be spawned foo.2 drops out of the runahead pool, and foo.2 does not exist until it is spawned when dad.2 succeeds.

So this is non-optimal in two ways:

  • foo.3 is unnecessarily delayed until after dad.2 has spawned foo.2
  • foo.3 would be unnecessarily prevented from existing if dad.2 failed and thus did not spawn foo.2at all

Probably not hard to fix - the auto-spawn mechanism, including at start-up, just needs to use the available sequence info better.

@hjoliver hjoliver added this to the cylc-8.0b0 milestone Aug 6, 2020
@hjoliver hjoliver self-assigned this Aug 6, 2020
@hjoliver hjoliver modified the milestones: cylc-8.0b0, cylc-8.0.0 Dec 1, 2020
@hjoliver hjoliver modified the milestones: cylc-8.0.0, cylc-8.x Aug 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants