Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Triggering parentless task in new flow caused it to spawn in later cycles #6258

Open
MetRonnie opened this issue Jul 23, 2024 · 4 comments
Open
Labels
question Flag this as a question for the next Cylc project meeting.
Milestone

Comments

@MetRonnie
Copy link
Member

MetRonnie commented Jul 23, 2024

Reproducible Example

[scheduler]
    allow implicit tasks = True
[scheduling]
    cycling mode = integer
    [[graph]]
        P1 = """
            end[-P1] => start
            start & compile => end
        """

[runtime]
    [[end]]
        script = false
cylc trigger example//1/compile --flow=new

causes 2/compile, 3/compile etc to spawn again.

image

Expected Behaviour

Only 1/compile should spawn.

Additional Context

Tested and reproduced all the way back in 8.2.6

@MetRonnie MetRonnie added the bug Something is wrong :( label Jul 23, 2024
@MetRonnie MetRonnie added this to the 8.3.4 milestone Jul 23, 2024
@hjoliver hjoliver removed the bug Something is wrong :( label Jul 24, 2024
@hjoliver
Copy link
Member

hjoliver commented Jul 24, 2024

Not a bug!

In any flow, including the original flow and "new" ones, parentless tasks continue on to future cycles by magically spawning next instances, since they have no parents to do it "on demand". If not for this you couldn't trigger a new flow that traverses the whole graph just like the original flow.

We probably do need an option to not do that, which would make this more of a feature request.

We have definitely discussed this before somewhere but I can't find the associated issue.

At the moment you can prevent the flow on with cylc set --out=expired --flow=2 on the next-cycle parentless task before triggering flow 2 (which means "don't run the task in flow 2 even if it becomes ready"). Then later you'll have to remove the manually-expired task in flow 2 - see #6221

Correction: for parentless tasks, this only prevents the one future instance (the manually expired one) from running. That's by design. If a parentless task expires or fails (say) that doesn't mean other instances beyond that cycle should not run.

So for the moment, you have to trigger with --flow=none to avoid flow-on to future cycles. Which obviously is no good if you do want flow-on within the same cycle.

@MetRonnie MetRonnie modified the milestones: 8.3.4, 8.3.x Aug 9, 2024
@MetRonnie MetRonnie added the question Flag this as a question for the next Cylc project meeting. label Aug 9, 2024
@oliver-sanders
Copy link
Member

I'm not sure what the question is here, what were you trying to achieve with this trigger?

  • Option 1: I wanted to rerun all tasks in the first cycle:
    • With reflow:
      • With reflow, we cannot presently set a "termination point" beyond which the new flow will not continue beyond.
      • So this approach would not have worked in this example because the end[-P1] => start would have caused the new flow to roll on into the next cycle.
      • If we didn't spawn the parentless tasks in future cycles, then the workflow would stall which would also be undesireable.
    • With remove:
      • The remove proposal will allow you to remove all tasks in the cycle, priming them for re-running (but not actually triggering them).
    • With group trigger:
      • The "group trigger" proposal extends remove functionality by also identifying and triggering the start tasks in the group.
  • Option 2: I just wanted to re-run this one task:
    • With reflow:
      • Use --flow=none
    • With remove:
      • Remove the old instance and trigger the task.
    • With group trigger:
      • Trigger the task (I think the proposal makes this the default behaviour).
  • Option 3: I wanted to re-run the workflow from the start.
    • With reflow:
      • You'll also need to trigger compile i.e cylc trigger example//1/compile --flow=new.
    • With remove:
      • Possible, but icky.
    • With group triggers:
      • Possible, might be icky, depends on the fine details of task matching.

IMO, the observed behaviour is correct for --flow=new. If we didn't spawn the tasks in future cycles, then the workflow would stall as a result of the trigger which would be highly undesirable.

Suggest ensuring that the intended use case is adequately covered by proposed interventions and closing this if it is?

@MetRonnie
Copy link
Member Author

MetRonnie commented Sep 4, 2024

The use case I can think of is "I want to re-run a chain in a particular cycle point, it will catch up with the blocked flow 1 and flow-merge, therefore it is unexpected that the task I triggered starts running in other cycle points"

@oliver-sanders
Copy link
Member

oliver-sanders commented Sep 4, 2024

The best match for that would be group trigger.

As I understand it (I haven't had the time to go through in detail yet):

cylc trigger workflow//1  # remove all tasks (and their outputs) and re-run from the start

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Flag this as a question for the next Cylc project meeting.
Projects
None yet
Development

No branches or pull requests

3 participants