Make rules Manager Update method no-op after Close #14366
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #14146
This is an alternative approach to PR #14175
I considered @machine424's initial suggestion to set
m.groups
to empty slice. However in caseUpdate
is called afterClose
we would end up with new rule groups running and that cannot be stopped usingStop
since it is not possible to callStop
twice. Actually the same thing happens with #14175 . The approach in this PR makes for a cleaner shutdown: no dangling go-routines that are stopped in an unpredictable way.As @roidelapluie mentioned, we don't have any case were we restart a stopped ruler, so I think this approach is acceptable.
Credits to @machine424 for providing the unit tests.
PS:
Another approach that I also considered, but that I don't think it is worth implementing is to make
Update
afterClose
return an error, (i.e.: make it illegal), but that would not solve the problem of the race condition during shutdown because it would produce misleading error messages in case there is a reload after SIGTERM.We would need to either:
Update
during reload in case the manager is already closed by adding a new method to itIsClosed
and a mutex that controls the access to the manager.Update
method that the reloader could use to distinguish between real errors and expected errors.