Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent task events manager logging the same message more than once #5562

Draft
wants to merge 18 commits into
base: 8.2.x
Choose a base branch
from

Conversation

wxtim
Copy link
Member

@wxtim wxtim commented Jun 1, 2023

Closes #5526

Deals with a bug was returning every message ever sent by a task on each poll.

(Draft pending further checks, to ensure that I haven't broken anything)

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included
  • CHANGES.md entry included if this is a change that can affect users
  • Update docs - this is a change to Cylc's logging.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@wxtim wxtim marked this pull request as draft June 1, 2023 09:07
@wxtim wxtim force-pushed the fix.task_message_repetition branch 4 times, most recently from 16f966d to 246c95e Compare June 6, 2023 10:49
@wxtim wxtim requested review from MetRonnie and hjoliver June 6, 2023 10:49
@wxtim wxtim self-assigned this Jun 6, 2023
@wxtim wxtim added bug Something is wrong :( small labels Jun 6, 2023
@wxtim wxtim added this to the cylc-8.1.5 milestone Jun 6, 2023
@wxtim wxtim marked this pull request as ready for review June 6, 2023 10:49
@MetRonnie MetRonnie linked an issue Jun 6, 2023 that may be closed by this pull request
Copy link
Member

@MetRonnie MetRonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is about more than just preventing duplicate logging. With your branch I still get duplicates in the UI

image

@wxtim wxtim marked this pull request as draft June 6, 2023 12:33
@wxtim wxtim force-pushed the fix.task_message_repetition branch from f1a8617 to add64fe Compare June 6, 2023 12:43
@wxtim wxtim marked this pull request as ready for review June 6, 2023 12:43
@wxtim
Copy link
Member Author

wxtim commented Jun 6, 2023

@MetRonnie - I think that I fixed it.

@wxtim wxtim marked this pull request as draft June 7, 2023 08:30
@wxtim
Copy link
Member Author

wxtim commented Jun 7, 2023

The original approach (adding messages to the activity log file and checking for them later seemed to break some of the workflow restart tests in ways that make me un-confident that they are safe changes.

Scraping the log is horrible.

Next thing to try: using the database - this isn't straightforward, because the database doesn't contain a table which would easily contain the task messages.

@wxtim wxtim force-pushed the fix.task_message_repetition branch from add64fe to 32930db Compare June 7, 2023 13:22
@wxtim wxtim requested a review from MetRonnie June 7, 2023 13:22
@wxtim wxtim marked this pull request as ready for review June 7, 2023 13:22
@wxtim wxtim marked this pull request as draft June 7, 2023 15:15
@wxtim wxtim force-pushed the fix.task_message_repetition branch from 47e109f to 7d9a83f Compare June 8, 2023 08:23
@wxtim wxtim marked this pull request as ready for review June 8, 2023 08:54
Copy link
Member

@MetRonnie MetRonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did some playing about with the UI

  1. Ran a workflow with a long-running task that does cylc message first thing
  2. Stopped the workflow with --now
  3. Restarted the workflow

The chip with the message did not appear on restart, whereas before it used to appear with "(polled)" appended

cylc/flow/task_events_mgr.py Outdated Show resolved Hide resolved
cylc/flow/task_events_mgr.py Outdated Show resolved Hide resolved
cylc/flow/task_events_mgr.py Outdated Show resolved Hide resolved
cylc/flow/task_events_mgr.py Outdated Show resolved Hide resolved
@wxtim wxtim requested a review from MetRonnie June 12, 2023 06:57
@hjoliver hjoliver modified the milestones: cylc-8.1.5, cylc-8.2.0 Jun 12, 2023
@oliver-sanders oliver-sanders modified the milestones: cylc-8.2.1, cylc-8.2.2 Aug 14, 2023
@wxtim
Copy link
Member Author

wxtim commented Aug 22, 2023

Update: closed that as it does not seem feasible without a major refactor

Not entirely surprised, but thank you for attempting it.

@wxtim wxtim force-pushed the fix.task_message_repetition branch 2 times, most recently from 57b0c98 to f12f521 Compare August 22, 2023 10:25
@wxtim wxtim force-pushed the fix.task_message_repetition branch from f12f521 to 001cc12 Compare August 22, 2023 10:56
@wxtim wxtim marked this pull request as ready for review August 22, 2023 10:58
@wxtim wxtim requested a review from MetRonnie August 22, 2023 10:58
@wxtim wxtim requested a review from MetRonnie August 23, 2023 10:16
Copy link
Member

@MetRonnie MetRonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got some suggestions for the tests and then I think this is good to go!

tests/functional/cylc-message/01-newline.t Outdated Show resolved Hide resolved
tests/functional/restart/58-removed-task.t Outdated Show resolved Hide resolved
tests/functional/restart/58-removed-task.t Outdated Show resolved Hide resolved
tests/functional/restart/58-removed-task.t Outdated Show resolved Hide resolved
Co-authored-by: Ronnie Dutta <61982285 [email protected]>
@wxtim wxtim requested a review from MetRonnie August 23, 2023 15:18
@wxtim
Copy link
Member Author

wxtim commented Sep 26, 2023

@oliver-sanders are you waiting on me to do something with this?

@oliver-sanders
Copy link
Member

oliver-sanders commented Sep 26, 2023

Kinda, but I need to take a closer look. I think this isn't skipping enough of the method, the problem is about more than just logging, e.g. it appears to running event handlers and spawning logic for duplicate messages? But I know that when you tried moving the return higher up the function it caused problems.

@oliver-sanders oliver-sanders modified the milestones: cylc-8.2.2, cylc-8.2.3 Oct 4, 2023
@oliver-sanders oliver-sanders marked this pull request as draft October 5, 2023 12:39
@oliver-sanders oliver-sanders modified the milestones: cylc-8.2.3, cylc-8.2.4 Nov 1, 2023
@oliver-sanders oliver-sanders self-assigned this Nov 8, 2023
@oliver-sanders oliver-sanders modified the milestones: cylc-8.2.4, cylc-8.x Jan 4, 2024
@wxtim
Copy link
Member Author

wxtim commented Oct 6, 2024

I've had another look at this recently, and the notes on Matt Shin's original logic says

       # Unhandled messages. These include:
       #  * general non-output/progress messages
       #  * poll messages that repeat previous results
       # Note that all messages are logged already at the top.

https://github.com/cylc/cylc-flow/pull/2157/files

So it looks like Matt intended to have it work more simply than this and it's broken at some point. I'm going to reexamine this. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is wrong :( small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Polling causes duplicate messages
4 participants