Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make rules Manager Update method no-op after Close #14366

Merged
merged 2 commits into from
Jun 29, 2024

Conversation

rapphil
Copy link
Contributor

@rapphil rapphil commented Jun 28, 2024

Fixes #14146

This is an alternative approach to PR #14175

I considered @machine424's initial suggestion to set m.groups to empty slice. However in case Update is called after Close we would end up with new rule groups running and that cannot be stopped using Stop since it is not possible to call Stop twice. Actually the same thing happens with #14175 . The approach in this PR makes for a cleaner shutdown: no dangling go-routines that are stopped in an unpredictable way.

As @roidelapluie mentioned, we don't have any case were we restart a stopped ruler, so I think this approach is acceptable.

Credits to @machine424 for providing the unit tests.


PS:
Another approach that I also considered, but that I don't think it is worth implementing is to make Update after Close return an error, (i.e.: make it illegal), but that would not solve the problem of the race condition during shutdown because it would produce misleading error messages in case there is a reload after SIGTERM.
We would need to either:

  • Avoid calling Update during reload in case the manager is already closed by adding a new method to it IsClosed and a mutex that controls the access to the manager.
  • Implement error types for the Update method that the reloader could use to distinguish between real errors and expected errors.

This has to be done because Close and Update methods are accessed concurrently.

Signed-off-by: Raphael Silva <[email protected]>
Signed-off-by: Raphael Silva <[email protected]>
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bwplotka bwplotka merged commit 4f2558d into prometheus:main Jun 29, 2024
25 checks passed
@machine424
Copy link
Collaborator

Thanks for working on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

panic: close of closed channel
3 participants