Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1.9.0 Grafana OnCall Plugin: Can't connect to Grafana Cloud OnCall #4943

Closed
tuminansen opened this issue Aug 28, 2024 · 5 comments
Closed

Comments

@tuminansen
Copy link

tuminansen commented Aug 28, 2024

What went wrong?

What happened:
I tried to connect the Grafana OnCall plugin (v1.9.0 as recommended in Grafana Community Slack, and also with newer versions) to a Grafana Cloud instance for OnCall, but ran into an issue. I set the GRAFANA_CLOUD_ONCALL_API_URL and GRAFANA_CLOUD_ONCALL_API_TOKEN environment variables, but they don't seem to have any effect.

Image

What did you expect to happen:

  • I expected the Grafana OnCall plugin to successfully connect to the Grafana Cloud OnCall instance using the provided backend URL and API token.

How do we reproduce it?

  1. In your self-hosted Grafana instance, Install Grafana OnCall Plugin
  2. Restart the instance with GRAFANA_CLOUD_ONCALL_API_URL and GRAFANA_CLOUD_ONCALL_API_TOKEN according to API URL and API TOKEN in your Grafana Cloud instance OnCall settings.
  3. In your self-hosted Grafana instance, insert the OnCall API URL (taken from Grafana Cloud instance) into the OnCall backend URL. In my case it's https://oncall-prod-us-central-0.grafana.net/oncall/
  4. Observe error

Grafana OnCall Version

v.1.9.0

Product Area

Auth, API, Other

Grafana OnCall Platform?

I use Grafana Cloud

User's Browser?

Chrome Version 128.0.6613.85 (Official Build) (arm64)

Anything else to add?

@tuminansen tuminansen added the bug Something isn't working label Aug 28, 2024
@tuminansen tuminansen changed the title Grafana OnCall Plugin: Can't connect to Grafana Cloud OnCall v1.9.0 Grafana OnCall Plugin: Can't connect to Grafana Cloud OnCall Aug 30, 2024
@youve
Copy link

youve commented Sep 2, 2024

I have a similar issue, reported it on the forums a few days ago, https://community.grafana.com/t/trying-to-connect-selfhosted-grafana-to-cloud-oncall/129985

The difference in my case is that I set the Oncall Backend URL in our self-hosted Grafana to point to our self-hosted Oncall-engine and it successfully connects to it, but the token for Grafana Cloud doesn't work even though it works with curl.

@tuminansen
Copy link
Author

We are still running into this error. Can anyone take a look, please?

@mderynck
Copy link
Contributor

mderynck commented Sep 6, 2024

Recently we made some changes to the way Grafana OnCall is initialized. Use 1.9.22, there were quite a few changes along the way from 1.9.0-1.9.22 to get things working.

  • If you are running Grafana 11 and newer you must have externalServiceAccounts feature toggle enabled.
    This has already been enabled in the docker compose files and helm charts in the oncall repo.
  • Plugin settings must be provided to the plugin using an API call if you are installing for the first time (Note: credentials and hostnames need to be adjusted for your configuration, stackId and orgId are expected to be the listed constants in a self-hosted configuration)
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://engine:8080/", "grafanaUrl":"http://grafana:3000/"}}'
  • Once settings are configured use this API call to install:
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/install'

Grafana OnCall should now be ready to use.
For additional troubleshooting see here

@tuminansen
Copy link
Author

Recently we made some changes to the way Grafana OnCall is initialized. Use 1.9.22, there were quite a few changes along the way from 1.9.0-1.9.22 to get things working.

  • If you are running Grafana 11 and newer you must have externalServiceAccounts feature toggle enabled.
    This has already been enabled in the docker compose files and helm charts in the oncall repo.
  • Plugin settings must be provided to the plugin using an API call if you are installing for the first time (Note: credentials and hostnames need to be adjusted for your configuration, stackId and orgId are expected to be the listed constants in a self-hosted configuration)
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' -H "Content-Type: application/json" -d '{"enabled":true, "jsonData":{"stackId":5, "orgId":100, "onCallApiUrl":"http://engine:8080/", "grafanaUrl":"http://grafana:3000/"}}'
  • Once settings are configured use this API call to install:
curl -X POST 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/install'

Grafana OnCall should now be ready to use. For additional troubleshooting see here

@mdyerynck This resolves the issue I was previously facing, but now I’m encountering a new challenge. Here’s the error I get in the UI:

Image

And this is what the API returns:

curl -X GET 'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/resources/plugin/status' | jq .
{
  "pluginConnection": {
    "settings": {
      "ok": true
    },
    "service_account_token": {
      "ok": true
    },
    "grafana_url_from_plugin": {
      "ok": true
    },
    "grafana_url_from_engine": {
      "ok": false,
      "error": "Not validated"
    },
    "oncall_api_url": {
      "ok": false,
      "error": "Not validated"
    },
    "oncall_token": {
      "ok": false,
      "error": "Unauthorized/Forbidden while accessing OnCall engine: /oncall/api/internal/v1/plugin/v2/status, status code: 403, check token"
    }
  },
  "license": "",
  "version": "",
  "currently_undergoing_maintenance_message": "",
  "api_url": ""
}

I’ve tried setting the ONCALL_API_TOKEN as both an environment variable when starting my Grafana instance and passing it in via the API call, like this:

curl -X POST \
'http://admin:admin@localhost:3000/api/plugins/grafana-oncall-app/settings' \
-H "Content-Type: application/json" \
-d '{
  "enabled": true,
  "jsonData": {
    "stackId": 5,
    "orgId": 100,
    "onCallApiUrl": "<ONCALL API URL>",
    "onCallApiToken": "<ONCALL API TOKEN>",
    "grafanaUrl": "http://localhost:3000/"
  }
}'

Looking through the code in the repo (https://github.com/grafana/oncall), it seems ONCALL_API_TOKEN is the token causing the issue, as mentioned here: https://github.com/grafana/oncall/tree/dev/tools/migrators#configuration

Image

@tuminansen
Copy link
Author

Apparently, you can't connect a plugin in OSS directly to a our cloud instance.
OSS installations are expected to run their own backend services

@tuminansen tuminansen closed this as not planned Won't fix, can't repro, duplicate, stale Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants