Hiya @blovett - Can you share what version of the monitoring chart you are using here? I mostly want to confirm because there are 2 versions of charts that we shipped for rancher-monitoring in 2.9.0 - so I want to confirm which of the two this issue happens on for you? Or if it happens on both versions?

Hiya @blovett - Can you share what version of the monitoring chart you are using here? I mostly want to confirm because there are 2 versions of charts that we shipped for rancher-monitoring in 2.9.0 - so I want to confirm which of the two this issue happens on for you? Or if it happens on both versions?

Hi. This is with rancher-monitoring-104.1.0+up57.0.3. I have not tried the other version yet. Sorry for leaving this out.

I installed the other version (104.0.0+up45.31.1) on a test cluster and it does not exhibit this behavior. Upgrading it to 104.1.0+up57.0.3 and it does. Downgrading, it doesn"t again.

@blovett - Greatly appreciate you clarifying the version and going as far as testing both!

We shipped two versions this release because we were upgrading the underlying components. So the issue seems to only exist in the newer version. From my quick test on my side it seems like the issue seems to be related to some missing or inaccessible web-assets. We"ll look into this further and see about getting this fixed.

Digging more into this - just going to dump a stream of w/e logging I can find:

Here"s the URL that is failing: /api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/chunk.8.8aa451110aebe6a6cc4f.js

From grafana container logs on rancher-monitoring-grafana- pod:

logger=context userId=0 orgId=1 uname= t=2024-08-14T14:21:07.503318251Z level=info msg="Request Completed" method=GET path=/chunk.8.8aa451110aebe6a6cc4f.js status=404 remote_addr=192.168.1.140 time_ms=1102 duration=1.102236101s size=45291 referer="/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/d/efa86fd1d0c121a26444b636a3f509a8/kubernetes-compute-resources-cluster?orgId=1&refresh=10s" handler=notfound

Upon doing a complete wip of site cache (via Application tab of devtools, "Clear Site Data") the UI loading issues no longer break the page visually. However the 404 for the JS file still happens and many graphs are incorrectly rendered. Of note - loading a different dashboard works fine.

I"ve determined the 404 isn"t the root cause of this issue. As the issue happens on dashboards where the 404 does not appear. The issue is actually related to this error in the JS console:

Uncaught (in promise) Error: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received

Edit: 🤔 So maybe not related to that uncaught error in promies...I"m observing the issue happening on render even before that error is logged.

This appears to be fixed in the pending rebase for 2.10 bumping to use 61.3.2 upstream.
I suspect that this will be backported to 2.9 and should resolve this issue at that time.

@mallardduck This issue is likely from here grafana/grafana#76180
The fix is included in Grafana v10.3.4, and it worked for me...

Internal ref: SURE-9001

@skanakal - I"ve tested and found we can upgrade to a higher version to fix as well. I have created initial PR to mirror the new versions we will need.

Still need a PR to patch existing affected chart versions to use new images.

Fix merged into dev-v2.9 branch under 104.1.2-rc.1+up57.0.3

Bug verification steps confirmed Grafana dashboards load correctly without issues with below version. The white screen issue appears resolved.

Details :
Rancher : v2.9.3-alpha5
cluster : v1.30.5+rke2r1
monitoring : 104.1.2-rc.1+up57.0.3
logging : 104.1.2-rc.1+up4.8.0

Steps to verify this bug:

• Access Grafana via Rancher UI.
• Select a dashboard, like "Kubernetes / Compute Resources / Cluster."
• Observe for any white screen error on loading the dashboard.

Attached screen shot

There is not the version 104.1.2-rc.1+up57.0.3 in charts, might better not to close the issue yet?

Now it"s changed to 104.1.2+up57.0.3