Thanos, Prometheus and Golang version used:

Starting in thanos v0.35.1, thanos-sidecar will hang waiting for Prometheus to start. But Prometheus isn't supposed to start until after thanos-sidecar. We use the Thanos sidecar --reloader.config-file and --reloader.config-envsubst-file features to populate Prometheus's config, which then allows Prometheus to start with a config that has had substitutions applied.

This is one of the few times in recent years I've had to rollback a Thanos release. Rolling back to v0.35.0 works.

Looking at the changelog, https://github.com/thanos-io/thanos/releases/tag/v0.35.1, it seems like #7323 introduced this.

Object Storage Provider:

What happened:

thanos-sidecar waits on Prometheus to start up.

What you expected to happen:

thanos-sidecar is not supposed to wait on Prometheus. It's responsible for templating the shared config.

How to reproduce it (as minimally and precisely as possible):

Start thanos-sidecar without Prometheus running. Observe that it hangs.

Full logs to relevant components:

Anything else we need to know: