Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

examples: extending workflow #6278

Merged
merged 1 commit into from
Aug 15, 2024

Conversation

oliver-sanders
Copy link
Member

  • Fix incorrect heading level
  • Add a note about scheduling tasks to run at the "stop after cycle point".

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included (or explain why tests are not needed).
  • Changelog entry included if this is a change that can affect users
  • Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@oliver-sanders oliver-sanders added small doc Documentation labels Aug 1, 2024
@oliver-sanders oliver-sanders added this to the 8.4.0 milestone Aug 1, 2024
@oliver-sanders oliver-sanders self-assigned this Aug 1, 2024
Copy link
Member

@hjoliver hjoliver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine, but I wonder if we should:

  • document that if you stop the workflow at any point before the final "stop task" (even after some, but not all, stop tasks have run) and restart with a later stop point, the original stop task branch will be truncated, and a new one "rebased" onto the new stop point
  • add explicit dependence on the main cycling graph so that the example stop tasks really don't start running until just before shutdown. As it stands, they will start running as soon as the runahead limit allows, concurrent with earlier cycles.

Both of these points should be kinda obvious, but IMO it doesn't hurt to make examples like this as explicit as possible.

@oliver-sanders
Copy link
Member Author

the original stop task branch will be truncated, and a new one "rebased" onto the new stop point

Sorry, I don't understand this.

I don't think the stop after cycle point config has any impact on the task pool does it? Restarting with a later stop CP doesn't appear to have any side effects?

@hjoliver
Copy link
Member

hjoliver commented Aug 5, 2024

I mean if the stop-task sub-graph has already started, but not finished, when the user stops or kills the workflow ... and then restarts it with a new stop-after-cycle-point.

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Aug 5, 2024

Does this not have the same impact as changing the stop cycle (note cycle not task) at runtime (e.g. via cylc stop or cylc reload)?

It just pushes the stop cycle back (as requested) but continue as normal with no graph changes?

@wxtim
Copy link
Member

wxtim commented Aug 5, 2024

I mean if the stop-task sub-graph has already started, but not finished, when the user stops or kills the workflow ... and then restarts it with a new stop-after-cycle-point.

What I think Hilary may be getting at, is that in the example the stop cycle point is hardcoded and the stop cycle point task will remain 3000 unless the user changes it in the config and reloads: It might be worth warning users that this case requires a change to the jinja2 value and a Cylc VR. If you do cylc play --stop-cycle-point=3002 then the stop cycle point graph will change.

The CLI centric route around this would be -s 'STOP_POINT=3000'

Part of me thinks that it'd not be unreasonable to provide the stop Cycle point as a template variable.

(None of the test failures look related to this change)

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Aug 5, 2024

What I think Hilary may be getting at, is that in the example the stop cycle point is hardcoded and the stop cycle point task will remain 3000 unless the user changes it in the config and reloads

The same applies to the initial/start/final cycle points?

Note, there is no "stop task" in this example, only a "stop cycle".

@hjoliver
Copy link
Member

hjoliver commented Aug 7, 2024

OK I'll try to explain myself better.

I'm just suggesting we should tell users the potential consequences of basing a sub-graph off of a dynamically set cycle point.

Original graph with stop_cycle = 3:

P1 = "foo[-P1] => foo => bar"
R1/3 = "bar => a => b => c & d". # {% set stop_cycle = 3 %}

image

But if I run this then shut it down while 3/b is running, i.e. part-way through the stop-cycle sub-graph, then I restart it with stop_cycle = 6 - what ends up running is this:

image

It might be obvious to us that the original stop-cycle sub-graph gets truncated and a whole new one gets based on cycle 6, but I'm not sure it'll be so obvious to all users.

@hjoliver
Copy link
Member

hjoliver commented Aug 7, 2024

I also think the example should have intercycle dependence (like mine above) and anchor the stop-cycle sub-graph to the main graph (like mine above), so that (in lieu of #5090) the stop-cycle sub-graph runs just before the workflow shuts down at the stop after cycle point, and not earlier than that (i.e. concurrent with earlier cycle points). That would almost certainly be wanted in a real use case.

@oliver-sanders
Copy link
Member Author

I'm just suggesting we should tell users the potential consequences of basing a sub-graph off of a dynamically set cycle point.

Gotcha.

The same thing applies to the final cycle point (e.g. R1/$) so should be expected, but fair enough, I'll add a note.

* Fix incorrect heading level
* Add a note about scheduling tasks to run at the
  "stop after cycle point".
@oliver-sanders
Copy link
Member Author

@hjoliver, done.

@hjoliver hjoliver merged commit b8be2e3 into cylc:master Aug 15, 2024
26 of 27 checks passed
@oliver-sanders oliver-sanders deleted the extending-workflow branch August 15, 2024 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc Documentation small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants