1

After a long one yesterday i ran into a problem with my Pulumi node.js program and have not been able to delete a simple stack tied to a couple AWS resources. I have tried all sorts of different ways to initialize the destroy but keep receiving a Terraform AWS Provider error.

In the past when I have ran into problems where i cant destroy a stack's resources I have used the following steps from the settings/options page in the Pulumi web app to recreate the configuration files.

from the pulumi app:

**Recovering Configuration

Deleting this stack or this stack's resources requires that you have a Pulumi.yaml file in the same directory.

If you no longer have access to the Pulumi program's source code, you can recreate the configuration files with the following commands: **

# Pulumi.yaml
echo "name: inlineNode" > Pulumi.yaml
echo "runtime: nodejs" >> Pulumi.yaml

# Pulumi.dev.yaml
pulumi stack select projectmikey/inlineNode/dev
pulumi config refresh

After running that I can usually follow with these two commands to destroy the stack resources and then remove it from pulumi...

pulumi destroy -s projectmikey/inlineNode/dev

and then

pulumi stack rm projectmikey/inlineNode/dev

However this time i am recieving the following error from Terraform.... uhhh....

Previewing destroy (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/projectmikey/inlineNode/dev/previews/<request-id>

     Type                    Name               Plan
 -   pulumi:pulumi:Stack     inlineNode-dev     delete
 -   ├─ aws:s3:BucketObject  index              delete
 -   ├─ aws:s3:BucketPolicy  bucketPolicy       delete
 -   └─ aws:s3:Bucket        s3-website-bucket  delete


Outputs:
  - websiteUrl: "s3-website-bucket-abcdef.s3-website-us-west-1.amazonaws.com"

Resources:
    - 4 to delete

Do you want to perform this destroy? yes
etrieving via all available methods. See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for workaround and implications. Errors: 2 errors occurred:
        * error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
        status code: 403, request id: <request-id>
        * failed getting account information via iam:ListRoles: InvalidClientTokenId: The security token included in the request is invalid.
        status code: 403, request id: <request-id>

Resources:

Duration: 2s

from the Pulumi web app

Changes:
 
    Type                    Name               Status                 Info
    pulumi:pulumi:Stack     inlineNode-dev                            
~   ├─ aws:s3:Bucket        s3-website-bucket  **refreshing failed**  
~   ├─ aws:s3:BucketObject  index              **refreshing failed**  
~   └─ aws:s3:BucketPolicy  bucketPolicy       **refreshing failed**  
 
Diagnostics:
  pulumi:pulumi:Stack (projectmikey/inlineNode/dev)
    error: update failed
 
  aws:s3:Bucket (s3-website-bucket)
    error: 1 error occurred:
    * error configuring Terraform AWS Provider: AWS account ID not previously found and failed retrieving via all available methods. See https://www.terraform.io/docs/providers/aws/index.html#skip_requesting_account_id for workaround and implications. Errors: 2 errors occurred:
    * error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: <request-id>
    * failed getting account information via iam:ListRoles: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: <request-id>


from the terraform link

skip_requesting_account_id - (Optional) Whether to skip requesting the account ID. Useful for AWS API implementations that do not have the IAM, STS API, or metadata API. When set to true and not determined previously, returns an empty account ID when manually constructing ARN attributes with the following:

so i have tried running

pulumi config set aws:skipRequestingAccountId true

i have also tried setting my aws accountId.

but nothing has worked to destroy this stack. The error is saying the security token is invalid, however i was never using a security token previously I was just using the aws access key id, aws secret access key, and aws region variables.

after reading suggestions from another post i have tried creating an aws session token, then unsetting my aws credentials, and then exporting them again with the token...

unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_REGION
unset AWS_SESSION_TOKEN

and then

aws configure
aws sts get-session-token

and then reentering my credentials...

export AWS_ACCESS_KEY_ID='xxx' &&
export AWS_SECRET_ACCESS_KEY='xxx' &&
export AWS_REGION='xxx' &&
export AWS_SESSION_TOKEN='xxx'

but still no luck....

The other weird part is after all of this i can create a second iteration of the stack with a different name and then destroy it without any issues so i would assume my AWS credentials are working correctly...

I am confused at what went wrong on this one any help would be greatly appreciated,

thanks in advance

Marko E
  • 13,362
  • 2
  • 19
  • 28
  • how did you initially set the credentials? Did you configured them as stack config options? – jaxxstorm Mar 18 '23 at 16:41
  • Initially i submitted them within my code using the pulumi automation api example here https://github.com/pulumi/automation-api-examples/tree/main/nodejs/inlineProgram-js. I have tried deleting them with this method as well. Here is the code ```javascript ... await stack.setConfig("aws:region", { value: "xxx" }); await stack.setConfig("aws:accessKey", { value: "xxx" }); await stack.setConfig("aws:secretKey", { value: "xxx" }); ... ``` – projectmikey Mar 18 '23 at 16:50

1 Answers1

0

From questions and comments I gather that:

  • You initially created a stack with Pulumi automation API and provided AWS creds directly from code to stack config with stack.setConfig("aws:xxx",...)
  • Creating another stack with new credentials works (you can both create and delete the stack)

Your problem may be that Pulumi continue to use the AWS credentials originally specified in the stack's config to delete existing resources, however for some reason these "old" credentials do not work anymore. Even setting AWS_* environment variables won't work as credentials in-config will take priority at runtime.

A few possible solutions:

  1. Update your stack config with new credentials:
    • Update credentials using working ones with commands like:
    pulumi config set "aws:accessKey" xxx
    pulumi config set "aws:secretKey" xxx
    
    • Then try to delete again:
    pulumi destroy
    
  2. If 1. doesn't work, update your stack's state directly to replace old (invalid) credentials by working ones:
    • Download state locally with
    pulumi stack export > state.json
    
    • Lookup for credentials in your state with accessKey and secretKey keywords and replace them with proper ones
    • Import back modified state
    pulumi stack import -f state.json
    
Pierre B.
  • 11,612
  • 1
  • 37
  • 58