-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Monitoring V2] no metrics from windows nodes available in Grafana when win_prefix_path is set on a windows cluster #79
Comments
Reproduced in the latest Monitoring chart. Screenshots and logs attached below for investigation. Seems like an issue with the Redeploying the wins client with windows-exporter itself doesn't seem to resolve the issue as it produces two more of the same exact logs:
check-wins-version logs
exporter-node logs
exporter-node-proxy logs
wins service logs (host)
named pipes (host)
|
Possible WorkaroundJust deploying rancher-wins-upgrader (e.g. re-initializing the wins service) seems to be an effective workaround to this issue. I'm not sure whether this is because the fix in wins @sowmyav27 once rc19 is cut with wins v0.1.1, can you retest this issue to see if that resolves it? |
@sowmyav27 & @aiyengar2 - We are doing some triage right now of issues in 2.6. Would you be able to give us more information about this? Is this fixed in the latest RC? And Arvind, how does the workaround look as a viable option? (You mentioned it was a possible workaround). |
@Jono-SUSE-Rancher I don't believe this is fixed in the latest RC. The core problem here seems to be that a Windows cluster without rancher-wins-upgrader deployed that mounts resources on a prefixPath (e.g. This issue appears to be resolved when the wins service is restarted and/or the wins config is refreshed, which is exactly what happens when you deploy I'm not sure why this restart is required so this needs to be investigated. The problem could be with the way we do bootstrapping on Windows nodes (e.g. how we set up the config service) or could require cutting a new wins release. Either way, this would be a Windows issue that is not particular to Monitoring (cc: @sirredbeard ). Currently, only Monitoring is impacted since only monitoring uses However, if we cannot prioritize this in 2.6, the workaround of expecting rancher-wins-upgrader to be deployed onto Windows clusters with prefixPath enabled sounds like a viable option to me. I think we should encourage customers to start using it anyways so that they can have declarative wins configs (i.e. an expectation that the upgrader chart exists would allow us to more easily cut wins releases in the future, if we need to add security fixes, golang bumps, or new features). @luthermonson @sirredbeard any thoughts here? Either way, if we prioritize the workaround, I think we should ensure that it is tested rigorously to ensure that we don't miss anything before suggesting it as the official solution to this issue. |
@sowmyav27 @aiyengar2 Could this be related to the fact that no metrics are available in grafana for k8s 1.21? |
@deniseschannon that should be unrelated. rancher/rancher#33465 is Monitoring V1; this is Monitoring V2. |
What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible):
on 2.5.8-rc18
Down
Expected Result:
Metrics from windows nodes should be available in Grafana.
Other details that may be helpful:
Environment information
rancher/rancher
/rancher/server
image tag or shown bottom left in the UI): 2.5.8-rc18The text was updated successfully, but these errors were encountered: