Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

acknowledgments randomly fail with EXACTLY_ONCE_ACKID_FAILURE #1951

Closed
matejdro opened this issue Jul 5, 2024 · 4 comments
Closed

acknowledgments randomly fail with EXACTLY_ONCE_ACKID_FAILURE #1951

matejdro opened this issue Jul 5, 2024 · 4 comments
Labels
api: pubsub Issues related to the googleapis/nodejs-pubsub API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@matejdro
Copy link

matejdro commented Jul 5, 2024 •
edited
Loading

Environment details

  • OS: Ubuntu Server 18.04.6 LTS
  • Node.js version: 16.13.1
  • @google-cloud/pubsub version: 4.5.0

Steps to reproduce

  1. Pull a message via subscriberClient.pull({ 1, returnImmediately: false, subscription: mySubscriptionId, })
  2. Ack this pulled message via subscriberClient.acknowledge({ ackIds: [message.ackId!], subscription: mySubscriptionId }) (at most several seconds after receiving, so message could not have expired yet)

Most of the time above works fine, but rarely, acknowledgement will fail with the following error:

INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed. Error: 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
     at callErrorFromStatus (/opt/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
     at Object.onReceiveStatus (/opt/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
     at Object.onReceiveStatus (/opt/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
     at Object.onReceiveStatus (/opt/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
     at /opt/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
     at processTicksAndRejections (node:internal/process/task_queues:78:11)
 for call at
     at ServiceClientImpl.makeUnaryRequest (/opt/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
     at ServiceClientImpl.<anonymous> (/opt/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
     at /opt/node_modules/@inovait/pubsub-pull-subscriber/node_modules/@google-cloud/pubsub/build/src/v1/subscriber_client.js:227:29
     at /opt/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
     at repeat (/opt/node_modules/google-gax/build/src/normalCalls/retries.js:80:25)
     at /opt/node_modules/google-gax/build/src/normalCalls/retries.js:119:13
     at OngoingCallPromise.call (/opt/node_modules/google-gax/build/src/call.js:67:27)
     at NormalApiCaller.call (/opt/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
     at /opt/node_modules/google-gax/build/src/createApiCall.js:112:30 {
   code: 3,
   details: 'Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.',
   metadata: Metadata {
     internalRepr: Map(2) {
       'google.rpc.errorinfo-bin' => [Array],
       'grpc-status-details-bin' => [Array]
     },
     options: {}
   },

   note: 'Exception occurred in retry method that was not classified as transient',
   statusDetails: [
     ErrorInfo {
       metadata: [Object],
       reason: 'EXACTLY_ONCE_ACKID_FAILURE',
       domain: 'pubsub.googleapis.com'
     }
   ],
   reason: 'EXACTLY_ONCE_ACKID_FAILURE',
   domain: 'pubsub.googleapis.com',
   errorInfoMetadata: {
     'BhYsXUZIUTcZCGhRDk9eIz81IChFEQMIFAV8fXBBUXVdWndoUQ0Zcn1gfGpdR1cDQlB8VlEbB2hObXUf1vWz9kthRUh7EgAEQFt4VlkZCWhdWnYOWAgx1P2MuMGC6z4Yfmv2p_GfO0ii4Pl8Zi09XxJLLD5-Ky5FQV5AEkwgG0RJUytDCypYEU4EISE-MD5FU0RQ': '
PERMANENT_FAILURE_INVALID_ACK_ID'
   }
 }

Relevant error seems to be EXACTLY_ONCE_ACKID_FAILURE, but we cannot find any documentation for this error anywhere.

We are not sure what causes this. After this happens the received message will remain "un-ackable". The only workaround we found so far is to let the message expire and get redelivered. Redelivered message can in most cases be acked normally.

Any ideas what could cause this?
@matejdro matejdro added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Jul 5, 2024
@product-auto-label product-auto-label bot added the api: pubsub Issues related to the googleapis/nodejs-pubsub API. label Jul 5, 2024
@feywind
Copy link
Collaborator

feywind commented Jul 5, 2024

I wonder if this has something to do with ack IDs being ack'd more than once. It's a server error being returned, though, so someone else might know more. @maheshgattani ?

@matejdro
Copy link
Author

matejdro commented Jul 9, 2024

We added a lot of logging and as far as we can tell, we never ack the message more than once. Also, this broken message will eventually expire and be re-delivered, which signifies that it was not acked.

@feywind
Copy link
Collaborator

feywind commented Sep 12, 2024

Yeah, I talked with the team and they said this should really be a support request, because otherwise we can't look into the logs and see what's going on with those errors. If you've got a support contract, try that.

I assume subscriberClient is the "v1" client, SubscriberClient. If yes, those calls (pull and acknowledge) are pretty much going directly to the service, so there shouldn't really be much involvement from the main library anyhow.

@feywind feywind closed this as completed Sep 12, 2024
@matejdro
Copy link
Author

We don't have a support contract. What else is there for us to do?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api: pubsub Issues related to the googleapis/nodejs-pubsub API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@matejdro @feywind