Page MenuHomePhabricator
Create Task
Maniphest T379687

Set up Grafana dashboard for monitoring Charts
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Follow up to T372081

  • make sure we have appropriate logging in place (@CDanis to do)
  • Set up Grafana dashboard for monitoring, starting with making a copy of the template (Charts to do)

We can get latency and also request success rates from different queries based on the one existing Prom metric.

  • Add alerts around latency ("under 5 seconds" perhaps), and one around 5xx result ratio

Details

SubjectRepoBranchLines /-
chart-renderer: use the metrics portoperations/deployment-chartsmaster 7 -1
chart-renderer: scrape metricsoperations/deployment-chartsmaster 1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

CCiufo-WMF created this task.Nov 12 2024, 9:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 12 2024, 9:41 PM
CCiufo-WMF removed aude as the assignee of this task.Nov 12 2024, 9:41 PM
CCiufo-WMF triaged this task as Medium priority.
CCiufo-WMF raised the priority of this task from Medium to High.
CCiufo-WMF updated the task description. (Show Details)
CCiufo-WMF mentioned this in T372081: Deploy Chart service in production.
CCiufo-WMF added a subscriber: CDanis.
CCiufo-WMF added a subscriber: aude.
CCiufo-WMF added a parent task: T379689: Deploy Charts to test2wiki Commons.Nov 12 2024, 10:47 PM
CCiufo-WMF set the point value for this task to 3.Nov 18 2024, 7:48 PM
CCiufo-WMF moved this task from Up Next to Sprint 11 on the Charts board.
CCiufo-WMF edited projects, added Charts (Sprint 11); removed Charts.
Seddon renamed this task from Test wiki post-deployment work to Set up Grafana dashboard for monitoring Charts.Nov 18 2024, 7:57 PM
Jdlrobson moved this task from Incoming to Blocked on the Charts (Sprint 11) board.Nov 19 2024, 6:48 PM
Jdlrobson moved this task from Blocked to Ready for Dev on the Charts (Sprint 11) board.
Jdlrobson moved this task from Ready for Dev to Doing on the Charts (Sprint 11) board.Tue, Nov 26, 1:12 AM
Jdlrobson subscribed.

I started the board here: https://grafana-rw.wikimedia.org/d/a8d73414-80d1-4965-9346-170eecaa04d6/charts-service?orgId=1
@CDanis is there some documentation around how to set this up?

Jdlrobson claimed this task.Tue, Nov 26, 1:18 AM
CCiufo-WMF moved this task from Sprint 11 to Sprint 13 on the Charts board.Mon, Dec 2, 7:44 PM
CCiufo-WMF edited projects, added Charts (Sprint 13); removed Charts (Sprint 11).
CCiufo-WMF moved this task from Incoming to Ready for Dev on the Charts (Sprint 13) board.
CCiufo-WMF moved this task from Ready for Dev to Doing on the Charts (Sprint 13) board.
CDanis claimed this task.Tue, Dec 3, 9:49 PM
gerritbot added a comment.Tue, Dec 3, 9:54 PM

Change #1100202 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] chart-renderer: scrape metrics

https://gerrit.wikimedia.org/r/1100202

gerritbot added a project: Patch-For-Review.Tue, Dec 3, 9:54 PM
gerritbot added a comment.Wed, Dec 4, 4:09 PM

Change #1100202 merged by jenkins-bot:

[operations/deployment-charts@master] chart-renderer: scrape metrics

https://gerrit.wikimedia.org/r/1100202

Maintenance_bot removed a project: Patch-For-Review.Wed, Dec 4, 4:30 PM
gerritbot added a comment.Wed, Dec 4, 6:15 PM

Change #1100516 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] chart-renderer: use the metrics port

https://gerrit.wikimedia.org/r/1100516

gerritbot added a project: Patch-For-Review.Wed, Dec 4, 6:15 PM
gerritbot added a comment.Wed, Dec 4, 7:31 PM

Change #1100516 merged by jenkins-bot:

[operations/deployment-charts@master] chart-renderer: use the metrics port

https://gerrit.wikimedia.org/r/1100516

Maintenance_bot removed a project: Patch-For-Review.Wed, Dec 4, 8:31 PM
Jdlrobson reassigned this task from CDanis to CCiufo-WMF.Thu, Dec 12, 7:16 PM
Jdlrobson moved this task from Doing to Ready for Signoff on the Charts (Sprint 13) board.
CDanis closed this task as Resolved.Thu, Dec 12, 7:45 PM

Basic dashboard with the most important high-level metrics: https://grafana.wikimedia.org/d/f10cba3c-086c-49b2-bb04-65d70b39969a/charts-high-level

Also fixed up the things that were blatantly broken on the microservice dashboard (although I'm still not super happy with the overall template/layout there)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits