Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud Secrets Manager often errors DEADLINE Exceeded #5043

Open
0xterran opened this issue Feb 13, 2024 · 7 comments
Open

Cloud Secrets Manager often errors DEADLINE Exceeded #5043

0xterran opened this issue Feb 13, 2024 · 7 comments

Comments

@0xterran
Copy link

0xterran commented Feb 13, 2024

Secrets Manager retrieved in 31.092 seconds

So slow!


Yes I have tried the quick fixes and google cloud docs & github/stackoverflow search.

Environment details

  • which product (packages/*): @google-cloud/secret-manager
  • OS: Linux Google Cloud Run
  • Node.js version: v18
  • npm version: 9.5
  • google-cloud-node version: 5.0.1

Steps to reproduce

About 25% of the time, cloud secrets manager will timeout when attempting to access a secret. I have logged avg duration to get a secret and its around 10-30 seconds on first attempt (and <1 sec thereafter). The code is run within firebase functions and cloud run on us-central1. All my secrets managers are set to auto-replication policy.

What is causing this issue and how can I fix it? It's severely critical.

The pattern I am using:

const secret = await getSecretFromGCloud() // this takes 10  seconds
axios.post(url, payload, headers)

Error:

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
at callErrorFromStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
at Object.onReceiveStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client.js:192:76)
at Object.onReceiveStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
at Object.onReceiveStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
at /workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/resolving-call.js:99:78
at processTicksAndRejections (node:internal/process/task_queues:77:11)
at runNextTicks (node:internal/process/task_queues:64:3)
at process.processTimers (node:internal/timers:509:9)

@0xterran
Copy link
Author

0xterran commented Feb 13, 2024

My most recent logs:

Secrets Manager retrieved in 11.064 seconds
Secrets Manager retrieved in 18.103 seconds 
Secrets Manager retrieved in 31.092 seconds

The pattern I am using:

const secret = await getSecretFromGCloud() // this takes 10  seconds
axios.post(url, payload, headers)

@nullobject
Copy link

Same problem here 😞

@0xterran
Copy link
Author

Anyone from gcloud know if this is just a region issue? It's severely critical

My settings:

  • no custom encryption (gcloud managed)
  • automatic region (invokations from us-central1)

@antongildebrand
Copy link

I have the same problem. Loading a secret takes ~10 seconds for me. The weird thing is that it's only slow for me, not my colleagues, even though we're on the same network, hardware etc.

@david1542
Copy link

I also seem to experience the same problem. I have an express.js app which tries to retrieve some secrets

@fwermus
Copy link

fwermus commented May 19, 2024

happening here

@fwermus
Copy link

fwermus commented May 19, 2024

how do you manage to use secrets in a cloud function in case of cold start?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants