One prometheus instance is a SPOF, and might not even be able to monitor all the instances. This task (likely going to be split into subtasks) is to investigate and build out a HA/scaled-up configuration, possibly using some of those tools:
Note that this task is purely about the ability to query data from multiple sources, data storage is still handled on individual Prometheus nodes. While Thanos Store looks promising, it requires an object store (such as swift) and is considered Future Work.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T205862 Expand modern metrics infrastructure coverage (2018-19 Q2 goal) | |||
| Resolved | colewhite | T183454 Deprovision Diamond collectors no longer in use | |||
| Resolved | MoritzMuehlenhoff | T210993 Deprecate Diamond collectors in Cloud VPS | |||
| Resolved | taavi | T336774 Current status of cloudmetrics and its components | |||
| Resolved | taavi | T326266 Remove the WMCS statsd/Graphite service | |||
| Open | dcaro | T313444 Streamline WMCS Alerting and Paging | |||
| Resolved | taavi | T317032 Remove Diamond? | |||
| Resolved | taavi | T264920 Grafana "cloud-vps-project-board" needs to be migrated from Graphite to Prometheus | |||
| Open | None | T194333 [Epic] Provide logging/metrics/monitoring SaaS for Cloud VPS tenants | |||
| Resolved | taavi | T266050 Build Prometheus service for use by all Cloud VPS projects and their instances | |||
| Resolved | taavi | T286301 Scale up metricsinfra prometheus beyond one Prometheus instance (Thanos/Cortex/similar) |
Cortex's multi tenancy seems to require separate Prometheus instances per "tenant". With that in mind I'm leaning towards using Thanos for scaling metricsinfra up, mainly because it's already used an tested on production and metricsinfra could possibly re-use parts of its puppetisation.
Change 806551 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:wmcs::metricsinfra::prometheus: enable thanos sidecar
Change 806552 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:metricsinfra: add thanos query
Change 806553 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:metricsinfra::haproxy: add thanos routing
Change 806551 merged by David Caro:
[operations/puppet@production] P:wmcs::metricsinfra::prometheus: enable thanos sidecar
Change 806552 merged by David Caro:
[operations/puppet@production] P:metricsinfra: add thanos query
Change 806553 merged by David Caro:
[operations/puppet@production] P:metricsinfra::haproxy: add thanos routing
Change 850629 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:metricsinfra: add thanos rule
Change 850629 merged by David Caro:
[operations/puppet@production] P:metricsinfra: add thanos rule