Inconsistent Reset/Removal of metadata.labels for serviceMonitor rook-ceph-mgr #14472
Labels
Comments
|
@mohitrajain Could this be the same issue as #14477? There is a PR in progress to fix that. |
|
@travisn The behavior is same as the issue mentioned. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is this a bug report or feature request?
Deviation from expected behavior:
The
metadata.labelsforserviceMonitorrook-ceph-mgrare being reset inconsistently. We have enabledmonitoringfor thecephClusterand configured labels accordingly. While the labels are consistently applied to theserviceMonitorrook-ceph-exporter, they are absent forrook-ceph-mgr.Sharing metadata for both services here:
Sharing label and monitoring config for CephCluster as:
It is impacting our monitoring as prometheus operator has a selector:
The image below demonstrates how the label reset on
rook-ceph-mgris impacting our monitoring.Expected behavior:
We expect the
serviceMonitorrook-ceph-mgrto consistently have labels present. The absence of these labels is impacting alerts and overall monitoring effectiveness.How to reproduce it (minimal and precise):
I'm not sure how this issue is triggered. I've noticed it in other clusters but haven't had the chance to investigate thoroughly. I suspect it might be related to the cluster reconciliation process. I'm sharing logs here for your reference, which correlate with the label reset.
I was reviewing the code to get some insights and noticed that the implementation of
applyCephExporterLabelsin exporter.go differs from mgr.go. Specifically, there is a difference highlighted in the following code segment :I'm not certain if this difference is responsible for the issue, but it is worth considering.
File(s) to submit:
cluster.yaml, if necessaryLogs to submit:
Operator's logs, if necessary
Crashing pod(s) logs, if necessary
To get logs, use
kubectl -n <namespace> logs <pod name>When pasting logs, always surround them with backticks or use the
insert codebutton from the Github UI.Read GitHub documentation if you need help.
Cluster Status to submit:
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthTo get the status of the cluster, use
kubectl rook-ceph ceph statusFor more details, see the Rook kubectl Plugin
Environment:
uname -a):rook versioninside of a Rook Pod): v1.14.3ceph -v): 18.2.2kubectl version): v1.27.9ceph healthin the Rook Ceph toolbox): HEALTH_OKThe text was updated successfully, but these errors were encountered: