Improve the Node/Worker Pool Overview dashboard #6595
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The query of the panel
"Average # of Pods on each Node in Worker Group(s) $worker_group"
was incorrect: it was off by 1.
The new query uses count instead of sum because actually we want to
count the pods.
When the 2 vectors are merged with the operator to enrich the series
with the worker group label, we should add 0 instead of 1 to avoid
manipulating the result.
Note that we used the new Prometheus feature to format this query:
```
avg by (label_worker_gardener_cloud_pool) (
count by (node) (kube_pod_info{type="shoot"})
on (node) group_right ()
sum by (label_worker_gardener_cloud_pool, node) (
0 * kube_node_labels{label_worker_gardener_cloud_pool=~"etcd. "}
)
)
```
The query of the panel
"Number of Nodes in Worker Group(s) $worker_group"
could be simplified.
The query of the panel
"Average Memory Usage of Nodes in Worker Group(s) $worker_group"
was slightly improved.
The query of the panel
"Average CPU Usage of Nodes in Worker Group(s) $worker_group"
was improved to exclude the iowait and steal CPU times.
The "Failed Connection Tracking Inserts" panel was missing the
rate function for the counter metric.
The queries of the "Overview of all nodes" table were also formatted
and improved in a similar way.
Co-authored-by: Wesley Bermbach <[email protected]>
Co-authored-by: Istvan Zoltan Ballok <[email protected]>
Co-authored-by: Jeremy Rickards <[email protected]>
gardener-prow
bot
added
area/monitoring
gardener-prow
bot
added
cla: yes
rickardsjp
approved these changes
Aug 30, 2022
ialidzhikov
reviewed
Aug 30, 2022
| @@ -553,51 553,51 @@ | |||
| ], | |||
| "targets": [ | |||
| { | |||
| "expr": "sum(rate(node_cpu_seconds_total{mode!=\"idle\"}[$__rate_interval])) by (node) on (node) group_right sum(kube_node_labels{label_worker_gardener_cloud_pool=~\"$worker_group\"}) by (label_worker_gardener_cloud_pool, node) - 1", | |||
| "expr": " sum by (node) (rate(node_cpu_seconds_total{mode!~\"idle|iowait|steal\"}[$__rate_interval]))\n on (node) group_right ()\n sum by (label_worker_gardener_cloud_pool, node) (\n 0 * kube_node_labels{label_worker_gardener_cloud_pool=~\"$worker_group\"}\n )", | |||
There was a problem hiding this comment.
Do we need the additional whitespace at the beginning of the expression (is it part of the fancy formatting or can it be removed)?
There was a problem hiding this comment.
The leading whitespace is part of the formatting.
This is the new Prometheus formatting feature: prometheus/prometheus#11039
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ialidzhikov, rickardsjp The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
gardener-prow
bot
added
the
approved
ialidzhikov
reviewed
Aug 31, 2022
| @@ -46,7 46,7 @@ | |||
| "steppedLine": false, | |||
| "targets": [ | |||
| { | |||
| "expr": "avg(sum(rate(node_cpu_seconds_total{mode!=\"idle\"}[$__rate_interval])) by (node) on (node) group_right sum(kube_node_labels{label_worker_gardener_cloud_pool=~\"$worker_group\"}) by (label_worker_gardener_cloud_pool, node) - 1) by (label_worker_gardener_cloud_pool)", | |||
| "expr": "avg by (label_worker_gardener_cloud_pool) (\n sum by (node) (rate(node_cpu_seconds_total{mode!~\"idle|iowait|steal\"}[$__rate_interval]))\n on (node) group_right ()\n sum by (label_worker_gardener_cloud_pool, node) (\n 0 * kube_node_labels{label_worker_gardener_cloud_pool=~\"$worker_group\"}\n )\n)", | |||
There was a problem hiding this comment.
There are also other occurrences of
node_cpu_seconds_total{mode!~\"idle\"}
Don't they need to be adapted to make sure that all dashboards show the same CPU usage value? Otherwise I expect some minor differences.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
How to categorize this PR?
/area monitoring
/kind enhancement
What this PR does / why we need it:
Improve the Node/Worker Pool Overview dashboard
The query of the panel
"Average # of Pods on each Node in Worker Group(s) $worker_group"
was incorrect: it was off by 1.
The new query uses count instead of sum because actually we want to
count the pods.
When the 2 vectors are merged with the operator to enrich the series
with the worker group label, we should add 0 instead of 1 to avoid
manipulating the result.
Note that we used the new Prometheus feature to format this query:
The query of the panel
"Number of Nodes in Worker Group(s) $worker_group"
could be simplified.
The query of the panel
"Average Memory Usage of Nodes in Worker Group(s) $worker_group"
was slightly improved.
The query of the panel
"Average CPU Usage of Nodes in Worker Group(s) $worker_group"
was improved to exclude the iowait and steal CPU times.
The "Failed Connection Tracking Inserts" panel was missing the
rate function for the counter metric.
The queries of the "Overview of all nodes" table were also formatted
and improved in a similar way.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
/cc @rickardsjp
Release note: