I don't think we preserve the held state once a task has left the pool (fact check this)

I think we do. Take this one:

[scheduler]
    UTC mode = True
    allow implicit tasks = True
[scheduling]
    initial cycle point = 20240101T00
    [[graph]]
        T-00 = """
a[-PT1H] => a => b?
b? => c
b:failed? => d
"""
[runtime]
    [[root]]
        script = sleep $((5   $RANDOM % 10))
    [[b]]
        post-script = """
if ((1   $RANDOM % 10  < 6 )); then
    false
fi
"""

if you hold both c and d, then one will disappear with the output of b, but if you trigger b until the other path is taken then the respective task will reappear held.

This is arguably the correct behaviour.

Interestingly the other task sticks around even with the change in output of b.. To be expected but possibly not desirable..

Perhaps we should test for retries before putting a killed task on hold?

I'm not sure your example applies to the issue @dwsutherland ?

I don't think we preserve the held state once a task has left the pool (fact check this) [*]

If you hold c and d, they get added to the scheduler's "future hold" list, to be held if/when they get spawned into n=0.

Retriggering b until the other output is generated will spawn the other downstream task into n=0, after which both of them will be in n=0 and held. Neither of them leave the task pool, to test the statement [*]

Having both b and c in the hold list despite them being on mutually exclusive branches is correct, because the user asked for it (and of course retriggering can cause the other branch to spawn).

Extending your scenario a bit (beyond what you described at least):

• manually trigger the held c, so it succeeds and leaves the pool
  • c is shown as succeeded and held in GUI, so the datastore does preserve the held state once a task has left the pool
• manually trigger b in a new flow until it succeeds and spawns c again
• the new-flow c (waiting in n=0) will still be held

So the behaviour is consistent with what the datastore shows.

But we should consider releasing the hold when a task is removed from n=0 so that the hold only applies to the next flow (and not all subsequent flows after that). Would users really want a task to remain held even if a new flow comes along later?