-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regression in thanos v0.35.1 #7400
Comments
My bad! Ill make the waiting asynchronous and use a channel to ping the servers that they can start, that way the reloader should run concurrently! Let me draft up a PR |
just found this when trying to use a chainguard image been fighting with it for a whole day not understanding why it never populated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanos, Prometheus and Golang version used:
Starting in thanos v0.35.1, thanos-sidecar will hang waiting for Prometheus to start. But Prometheus isn't supposed to start until after thanos-sidecar. We use the Thanos sidecar
--reloader.config-file
and--reloader.config-envsubst-file
features to populate Prometheus's config, which then allows Prometheus to start with a config that has had substitutions applied.This is one of the few times in recent years I've had to rollback a Thanos release. Rolling back to v0.35.0 works.
Looking at the changelog, https://github.com/thanos-io/thanos/releases/tag/v0.35.1, it seems like #7323 introduced this.
Object Storage Provider:
What happened:
thanos-sidecar waits on Prometheus to start up.
What you expected to happen:
thanos-sidecar is not supposed to wait on Prometheus. It's responsible for templating the shared config.
How to reproduce it (as minimally and precisely as possible):
Start thanos-sidecar without Prometheus running. Observe that it hangs.
Full logs to relevant components:
Anything else we need to know:
The text was updated successfully, but these errors were encountered: