What went wrong?

What happened:

• I have installed grafana oncall using helm chart in

Log in pod oncall error is
2024-08-15 06:26:46 source=engine:app google_trace_id=none logger=root inbound latency=0.021681 status=403 method=POST path=/api/internal/v1/plugin/status user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:129.0) Gecko/20100101 Firefox/129.0 content-length=0 slow=0
2024-08-15 06:26:46 source=engine:app google_trace_id=none logger=django.request Forbidden: /api/internal/v1/plugin/status
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-incident-app/settings
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.021797078021336347 status=404 method=GET url=https://grafana.com.kepo.ninja/api/plugins/grafana-incident-app/settings slow=0
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-labels-app/settings
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.02221202797954902 status=404 method=GET url=https://grafana.com/api/plugins/grafana-labels-app/settings slow=0
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root inbound latency=1.782533 status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:129.0) Gecko/20100101 Firefox/129.0 content-length=0 slow=0
2024-08-15 03:35:35 source=engine:uwsgi status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install latency=1.783181 google_trace_id=- protocol=HTTP/1.1 resp_size=446 req_body_size=0
lamboksilitonga@Lambok-Silitonga ~ % 2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-incident-app/settings
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.021797078021336347 status=404 method=GET url=https://grafana.com/api/plugins/grafana-incident-app/settings slow=0
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=apps.grafana_plugin.helpers.client Error connecting to api instance 404 Client Error: Not Found for url: https://grafana.com/api/plugins/grafana-labels-app/settings
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root outbound latency=0.02221202797954902 status=404 method=GET url=https://grafana.com/api/plugins/grafana-labels-app/settings slow=0
2024-08-15 03:35:35 source=engine:app google_trace_id=none logger=root inbound latency=1.782533 status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:129.0) Gecko/20100101 Firefox/129.0 content-length=0 slow=0
2024-08-15 03:35:35 source=engine:uwsgi status=201 method=POST path=/api/internal/v1/plugin/self-hosted/install latency=1.783181 google_trace_id=- protocol=HTTP/1.1 resp_size=446 req_body_size=0x
zsh: parse error near `)'

Here is when access the grafana domain from oncall pod
grafana-oncall-engine-6ff6987857-rq4t2:/etc/app$ wget --header="Authorization: Bearer $token-grafana" https://grafana.com/api/health
Connecting to grafana.com (34.87.111.98:443)
saving to 'health'
health 100% |***************************************************************************************************************| 101 0:00:00 ETA
'health' saved
grafana-oncall-engine-6ff6987857-rq4t2:/etc/app$ ls
Dockerfile grpcio-1.64.1-cp312-cp312-linux_aarch64.whl settings
apps health static
celery_with_exporter.sh manage.py tox.ini
collected_static pyproject.toml type_stubs
common requirements-dev.in uwsgi.ini
config_integrations requirements-dev.txt wait_for_test_mysql_start.sh
conftest.py requirements.in
engine requirements.txt
grafana-oncall-engine-6ff6987857-rq4t2:/etc/app$ cat health
{
"commit": "b037f37bfe5858b91282557c126b27f7bac9246",
"database": "ok",
"version": "10.4.5"

grafana-584c96b4f8-d2k7g:/usr/share/grafana$ curl -v http://observability-endpoint.nonprod.tiket.tools:20005/

• Host observability-endpoint.nonprod.tiket.tools:20005 was resolved.
• IPv6: (none)
• IPv4: 192.168.69.200
• Trying 192.168.69.200:20005...
• Connected to observability-endpoint.nonprod.tiket.tools (192.168.69.200) port 20005

GET / HTTP/1.1
Host: observability-endpoint.nonprod.tiket.tools:20005
User-Agent: curl/8.5.0
Accept: /

< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< X-Frame-Options: DENY
< Content-Length: 2
< Vary: Origin
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Cross-Origin-Opener-Policy: same-origin
<

• Connection #0 to host observability-endpoint.nonprod.tiket.tools left intact

What did you expect to happen:

• Grafana can connect to oncall (Version v1.8.9)

How do we reproduce it?

Get log from pod grafana oncall engine

Grafana OnCall Version

v1.8.9

Product Area

Auth, API, Helm/Kubernetes/Docker

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

No response