-
Notifications
You must be signed in to change notification settings - Fork 436
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cloud tasks: Deadline exceeded -- retry settings not working #6814
Comments
Hi @mat105 A couple of things to note here. As for today, we can't retry the exceptions which are of the type Another thing that I would recommend is to use the retryable codes as constants like so: There is an experimental option that we released recently in GAX version 1.25.0 where you can pass your own custom retry function, using that you should be able to take care of this use case. However, please know that this behaviour might change in any future release as this is still experimental.
Another note of caution, please make sure you limit the retry behaviour by your own self checks as you might go into a loop of retrying which might incur cost. |
We're also hitting this issue. Is it possible to increase the 10 second timeout? |
It is possible that the From what I can see in the RetrySettings and RetryMiddleware in GAX, the only time we throw the @LukeAbell Can you verify that the retry is in fact taking place, though? |
I am facing a similar issue but is seems to be related to adding tasks in the queue, not the actual processing. When adding a lot of tasks, it can take more than 10s to finish the call, so the return is deadline exceeded. |
Hi @Rodrigo-JM In my tests I tried to create tasks serially in the order of a 1000 and I didn't get any errors. Then I tried simulating a post request of over a 100MB payload and I created 100s of such tasks serially and I wasn't able to recreate this issue. I wish to understand that since So, if there is a specific kind of task that is taking over 10 seconds to be added to the queue, then I would love some help to replicate this behaviour. Meanwhile, @LukeAbell could you let me know if the retries are being triggered at all after passing the Thanks. |
I using GCP's JS SDK for Cloud Tasks. The problem is related to a change they made in the service responsible for enqueuing tasks . Now to solve this issue we have to select the older version by setting:
Some other implementation I did was enqueuing in batches, but not sure it was needed. |
I have this code, however the retrySettings doesn't seem to be working (however they get validated), i keep getting constant DEADLINE_EXCEEDED errors, apparently there is a 10 second timeout somewhere, how can i fix this problem??
Environment details
Steps to reproduce
Code example
Error
Stack trace
The text was updated successfully, but these errors were encountered: