Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client-secrets-manager - ERROR: getApplicationSecrets The security token included in the request is invalid. #6355

Open
3 tasks done
geoffcorey opened this issue Aug 5, 2024 · 13 comments
Assignees
Labels
bug This issue is a bug. p2 This is a standard priority issue response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days.

Comments

@geoffcorey
Copy link

Checkboxes for prior research

Describe the bug

using aws sso login and running the application using @aws-sdk/client-secrets-manager version 3.623.0 we get the error.

If we drop back to 3.441.0 we are able to retrieve the AWS secrets

SDK version number

3.623.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

20.11.1

Reproduction Steps

OS/X Sonoma 14.5
Node 20.11.1


~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ cat package.json
{
  "name": "awsbug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-secrets-manager": "^3.623.0"
  }
}

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ cat index.js
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');
const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'my-key'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    logger: null
  });

  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  const response = await client.send(command);
  return response.secretString;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)

❯ node index.js
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:21:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '38866c6c-e16e-4dbf-bbf5-1a6de4fc4fc0',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

Change package to 3.441.0


~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ vi package.json

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 20s
❯ rm package-lock.json

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ rm -fr node_modules

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ npm i

added 78 packages, and audited 79 packages in 2s

2 packages are looking for funding
  run `npm fund` for details

3 high severity vulnerabilities

To address all issues, run:
  npm audit fix --force

Run `npm audit` for details.

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 2s
❯ cat package.json
{
  "name": "awsbug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-secrets-manager": "3.441.0"
  }
}

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ node index.js
success

Observed Behavior

upgrading package to latest from 3.441.0 breaks retrieval of secrets when using aws sso login on local machine running OS/X

Expected Behavior

Should work just like 3.441.0

Possible Solution

Don't ever upgrade?

Additional Information/Context

No response

@geoffcorey geoffcorey added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 5, 2024
@aBurmeseDev
Copy link
Member

Hi @geoffcorey - thanks for reaching out.

The error "UnrecognizedClientException: The security token included in the request is invalid" suggests that the temporary security credentials used to authenticate the request to AWS Secrets Manager are either invalid or have expired.

Since you mentioned that you are using AWS SSO for authentication, it's likely that the temporary credentials obtained through the AWS SSO session have expired.

To help diagnose and resolve the issue, it would be helpful if you could provide more details on how you have configured the AWS SSO session and the AWS profile in your application. Specifically, please share the following information:

  • How are you initializing and configuring the AWS SSO session in your application?
  • If you're using the AWS CLI, please share the command(s) you're running to initiate the AWS SSO session and configure the AWS profile.
  • If you're using the AWS SDK for JavaScript, please share the code snippet where you're configuring the AWS credentials using the fromSSO method from the @aws-sdk/credential-providers package.
  • Are you setting any environment variables or using any external configuration files to configure the AWS credentials or the AWS SSO session?
  • What is the expiration time or duration of the AWS SSO session you're using?

Providing these details will help us better understand your setup and configuration, which can assist in identifying the root cause of the issue and suggesting appropriate solutions.

Please refer to the AWS documentation links provided below for guidance on configuring AWS SSO with the AWS CLI and the AWS SDK for JavaScript (v3).
AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html
JavaScript SDK v3: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-credential-providers/#fromsso

@aBurmeseDev aBurmeseDev added response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Aug 6, 2024
@aBurmeseDev aBurmeseDev self-assigned this Aug 6, 2024
@geoffcorey
Copy link
Author

I call aws sso login from the command line. If I use the latest package version the app will not connect. if I use the 441 version it does.

version of the cli client

❯ aws --version
aws-cli/2.17.22 Python/3.11.9 Darwin/23.5.0 source/x86_64

We have not moved to v3 aws-cli due to some issue. I will upgrade tomorrow and upgrade the package and see if it will work.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Aug 7, 2024
@geoffcorey
Copy link
Author

geoffcorey commented Aug 7, 2024

So I am running the latest awscli client already and it is configured for SSO. Am I suppose to do something different for

const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');
const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'my-key'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    logger: null
  });

  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  const response = await client.send(command);
  return response.secretString;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

Because the latest fails with sso login and version 3.441.0 does not

@aBurmeseDev
Copy link
Member

Thanks for sharing additional info.

I call aws sso login from the command line. If I use the latest package version the app will not connect. if I use the 441 version it does

I’m a bit unclear on the details, especially since you also mentioned being unable to connect with aws sso login. Could you clarify what happens when you attempt to log in? Normally, you should see a 'Successfully logged into' message in your command line. You can refer to the documentation for more details.

$ aws sso login --profile my-dev-profile
SSO authorization page has automatically been opened in your default browser. 
Follow the instructions in the browser to complete this authorization request.
Successfully logged into Start URL: https://my-sso-portal.awsapps.com/start

Let’s say you were able to successfully configure and login SSO with CLI, you can leverage these credentials in your JavaScript SDK code. Here's an example using credential providers: can use these credential providers in your JavaScript SDK code. See here.

const client = new FooClient({ credentials: fromSSO({ profile: "my-sso-profile" })});

@aBurmeseDev aBurmeseDev added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Aug 7, 2024
@geoffcorey
Copy link
Author

I successfully login using aws sso login

If I execute the code I posted above using 3.441.0 of @aws-sdk/client-secrets-manager the code will pull the secrets successfully. If I use 3.623.0 and execute the code it throws and error

❯ node index.js
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:21:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '38866c6c-e16e-4dbf-bbf5-1a6de4fc4fc0',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Aug 9, 2024
@geoffcorey
Copy link
Author

Any insights?

@geoffcorey
Copy link
Author

Updates?

@aBurmeseDev
Copy link
Member

aBurmeseDev commented Aug 16, 2024

Hi @geoffcorey - apologies for the delay.

It is peculiar that the issue manifests itself exclusively in version 3.623.0. I recently attempted to replicate the error using the same version, but I was unable to reproduce it. Have you had the opportunity to explore the SDK credential providers? If leveraging those providers resolves the problem, we may be able to pinpoint the underlying cause more effectively.

Let’s say you were able to successfully configure and login SSO with CLI, you can leverage these credentials in your JavaScript SDK code. Here's an example using credential providers: can use these credential providers in your JavaScript SDK code. See here.

I'd also like to look at request logs, you can retrieve those by adding middleware stack to your client calls. Please share the logs from both versions.

const client = new DynamoDBClient({ region: "us-west-2" });

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

await client.listTables({});

Repro attempt

  • configure sso
$ aws configure sso
[profile my-dev-profile]
sso_session = my-sso
sso_account_id = 123456789011
sso_role_name = readOnly
region = us-west-2
output = json
  • sso login with --profile
$ aws sso login --profile my-dev-profile
SSO authorization page has automatically been opened in your default browser. 
Follow the instructions in the browser to complete this authorization request.
Successfully logged into Start URL: https://my-sso-portal.awsapps.com/start
  • SDK Code
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager"; 
import { fromSSO } from "@aws-sdk/credential-providers";

const client = new SecretsManagerClient({
   credentials: fromSSO({ profile: "my-dev-profile" }) }   
   region: "us-west-2";
});
const input = {
  SecretId: "STRING_VALUE", 
};
const command = new GetSecretValueCommand(input);
const response = await client.send(command);

@aBurmeseDev aBurmeseDev added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Aug 16, 2024
@geoffcorey
Copy link
Author

geoffcorey commented Aug 19, 2024

client.listTables isn't a valid call. Cut-n-paste error? Anyway, here is the two runs. Had someone on the infrastructure team search cloudtrail and did not see that request id in the logs.


~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ cat index.js
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');
const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'mykey'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    logger: null
  });
  client.middlewareStack.add(
    (next, context) => async (args) => {
      console.log("AWS SDK context", context.clientName, context.commandName);
      console.log("AWS SDK request input", args.input);
      const result = await next(args);
      console.log("AWS SDK request output:", result.output);
      return result;
    },
    {
      name: "MyMiddleware",
      step: "build",
      override: true,
    }
  )
  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  //await client.listTables({});
  const response = await client.send(command);
  return response;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ npm i && node index.js

added 78 packages, and audited 79 packages in 2s

2 packages are looking for funding
  run `npm fund` for details

3 high severity vulnerabilities

To address all issues, run:
  npm audit fix --force

Run `npm audit` for details.
AWS SDK context SecretsManagerClient GetSecretValueCommand
AWS SDK request input { SecretId: 'PA-Dev-key' }
AWS SDK request output: {
  '$metadata': {
    httpStatusCode: 200,
    requestId: '26e91637-e8f2-4da4-bbc7-7fada6b9c934',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  ARN: 'arn:aws:secretsmanager:us-west-2:169560604650:secret:PA-Dev-key-LefBMG',
  CreatedDate: 2024-07-18T16:21:10.776Z,
  Name: 'PA-Dev-key',
  SecretString: `{<REDACTED>}`,
  VersionId: '41bf8a9c-788a-4995-80f7-256377639827',
  VersionStages: [ 'AWSCURRENT' ]
}
success

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 3s
❯ vi package.json

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 10s
❯ rm package-lock.json && rm -fr node_modules && npm i && cat package.json && node index.js


added 77 packages, and audited 78 packages in 2s

2 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
{
  "name": "awsbug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-secrets-manager": "3.632.0"
  }
}
AWS SDK context SecretsManagerClient GetSecretValueCommand
AWS SDK request input { SecretId: 'PA-Dev-key' }
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:851:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/index.js:20:22
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:35:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '9a6cc38d-f322-4eac-96e5-2eb7207de45a',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

Node.js v20.11.1

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 2s

@geoffcorey
Copy link
Author

Also did a new aws configure and did a new profile. SSO login successful --profile with the new profile. Code still reports the same thing as above.


Successfully logged into Start URL: https://d-9a6716c0d7.awsapps.com/start#/

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 6s
❯ node index.js
AWS SDK context SecretsManagerClient GetSecretValueCommand
AWS SDK request input { SecretId: 'PA-Dev-key' }
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:851:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/index.js:20:22
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:35:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: 'ca8cf1cd-fafb-408f-b671-f1112419a838',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

Node.js v20.11.1

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)

@geoffcorey
Copy link
Author

@aBurmeseDev ok, so apparently if I am going to use SSO I have to pass it explicitly in the new version. In the 3.441.0 version it would figure it out. So doesn't seem functionally backwards compatible. Here is the work around.

const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

const { fromSSO } = require('@aws-sdk/credential-provider-sso');

const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'my-key'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    credentials: fromSSO(),
    logger: null
  });
  client.middlewareStack.add(
    (next, context) => async (args) => {
      console.log("AWS SDK context", context.clientName, context.commandName);
      console.log("AWS SDK request input", args.input);
      const result = await next(args);
      console.log("AWS SDK request output:", result.output);
      return result;
    },
    {
      name: "MyMiddleware",
      step: "build",
      override: true,
    }
  )
  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  //await client.listTables({});
  const response = await client.send(command);
  return response;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

@geoffcorey
Copy link
Author

Any thoughts on why it isn't backward compatible?

@geoffcorey
Copy link
Author

Additional notes. Installed a brand new machine with awscli via brew and this still fails without the fromSSO()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p2 This is a standard priority issue response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days.
Projects
None yet
Development

No branches or pull requests

2 participants