Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runahead limit not being cleared #4817

Closed
oliver-sanders opened this issue Apr 12, 2022 · 5 comments
Closed

runahead limit not being cleared #4817

oliver-sanders opened this issue Apr 12, 2022 · 5 comments
Labels
bug Something is wrong :(

Comments

@oliver-sanders
Copy link
Member

(see also #4037)

I have seen running ("naturally triggered") tasks appear in the GUI as runahead limited. We are probably missing a data store update somewhere causing the runahead status to get out of date. These tasks were probably both internally queued and runahead limited at the same time which might possibly be the cause of the issue?

Pull requests welcome!
This is an Open Source project - please consider contributing a bug fix
yourself (please read CONTRIBUTING.md before starting any work though).

@oliver-sanders oliver-sanders added the bug Something is wrong :( label Apr 12, 2022
@oliver-sanders oliver-sanders added this to the cylc-8.0rc4 milestone Apr 12, 2022
@hjoliver
Copy link
Member

Investigating...

@hjoliver hjoliver self-assigned this Apr 13, 2022
@hjoliver
Copy link
Member

I've failed to reproduce this - do you have the workflow you saw it in?

These tasks were probably both internally queued and runahead limited at the same time which might possibly be the cause of the issue?

Pretty sure that's not supposed to happen. Tasks should get queued only on release from runahead.

@oliver-sanders
Copy link
Member Author

do you have the workflow you saw it in?

Yes, but I don't think it will help much (it's a beast).

Here's a GUI screenshot showing a running task with a running job that is in the runahead state:

MicrosoftTeams-image

If I see this in the wild I will try to grab some graph GraphQL output and test in Tui to check whether the issue is definitely data store side.

These tasks were probably both internally queued and runahead limited at the same time which might possibly be the cause of the issue?

Pretty sure that's not supposed to happen.

This was more of a "steps to reproduce" than a description of what happened at the implementation level. I meant that the task could be considered to be held back by both runahead limit and (likely multiple) queues, perhaps the interaction between the two could be the root cause.

@oliver-sanders oliver-sanders changed the title data store: runahead limit not being cleared runahead limit not being cleared May 11, 2022
@hjoliver
Copy link
Member

(might be fixed by recent changes: #4906 ?)

@hjoliver hjoliver modified the milestones: cylc-8.0rc4, cylc-8.0.0 Jul 5, 2022
@oliver-sanders oliver-sanders modified the milestones: cylc-8.0.0, cylc-8.0.1 Jul 15, 2022
@oliver-sanders oliver-sanders modified the milestones: cylc-8.0.1, 8.0.2 Aug 16, 2022
@oliver-sanders oliver-sanders modified the milestones: cylc-8.0.2, cylc-8.0.3 Sep 12, 2022
@oliver-sanders oliver-sanders modified the milestones: cylc-8.0.3, cylc-8.0.4 Oct 12, 2022
@oliver-sanders oliver-sanders modified the milestones: cylc-8.0.4, cylc-8.1.x Dec 6, 2022
@oliver-sanders
Copy link
Member Author

This hasn't been seen in the wild for a while so closing this issue. Please re-open if this is spotted again.

@oliver-sanders oliver-sanders removed this from the cylc-8.1.x milestone Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is wrong :(
Projects
None yet
Development

No branches or pull requests

2 participants