Instead of referring an existing AWS S3 bucket, Cloud Formation is trying to create the bucket

Question

I'm stuck in a weird issue. I have created an AWS S3 bucket using following cloud formation template:-

AWSTemplateFormatVersion: '2010-09-09'
Metadata: 
  License: Unlicensed
Description: >
  This template creates a global unique S3 bucket in a specific region which is unique. 
  The bucket name is formed by the environment, account id and region

Parameters:
  # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html
  Environment:
    Description: This paramenter will accept the environment details from the user
    Type: String
    Default: sbx
    AllowedValues:
      - sbx
      - dev
      - qa
      - e2e
      - prod
    ConstraintDescription: Invalid environment. Please select one of the given environments only

Resources:
  # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html
  MyS3Bucket:
      Type: AWS::S3::Bucket
      DeletionPolicy: Retain
      Properties: 
        BucketName: !Sub 'global-bucket-${Environment}-${AWS::Region}-${AWS::AccountId}' # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html
        AccessControl: Private                
        LoggingConfiguration:
          DestinationBucketName: !Ref 'LoggingBucket'
          LogFilePrefix:  'access-logs'
        Tags:
          - Key: name
            Value: globalbucket
          - Key: department
            Value: engineering
  LoggingBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: !Sub 'global-loggings-${Environment}-${AWS::Region}-${AWS::AccountId}'
      AccessControl: LogDeliveryWrite      

Outputs:
  GlobalS3Bucket:
    Description: A private S3 bucket with deletion policy as retain and logging configuration
    Value: !Ref MyS3Bucket
    Export:
      Name: global-bucket

If you note in the template above then I'm exporting this S3 bucket in the Outputs section by the name called global-bucket.

Now, my intention is to refer to this existing bucket going forward in my AWS account whenever any new resource like Lambda, etc wants an S3 bucket. Here is an example using AWS SAM (Serverless Application Model), I'm trying to create an AWS Lambda and trying to refer to this existing S3 bucket using property !ImportValue and the export name as global-bucket as shown below:-

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  hellolambda

  Sample SAM Template for hellolambda
  
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Events:
        HelloLambdaEvent:
         Type: S3
         Properties:
            Bucket: !Ref SrcBucket
            Events: s3:ObjectCreated:*
  SrcBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !ImportValue global-bucket

Now, the problem is when I execute the command like sam build and then sam deploy --guided and select the same region (where my previous CloudFormation stack output is present) then I get the following error:-

global-bucket-sbx-ap-southeast-1-088853283839 already exists in stack arn:aws:cloudformation:ap-southeast-1:088853283839:stack/my-s3-global-bucket/aabd20e0-f57d-11ea-80bf-06f1487f6a64

The screenshot below:-

The problem is AWS CloudFormation is trying to create the S3 bucket rather than referring to the existing one.

But, if I try to update this SAM template like and then execute sam deploy, I get the following error:-

Waiting for changeset to be created.. Error: Failed to create changeset for the stack: my-lambda-stack, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [HelloWorldFunction] is invalid. Event with id [HelloLambdaEvent] is invalid. S3 events must reference an S3 bucket in the same template.

I'm blocked by both ends. I would really appreciate it if someone can assist to guide me writing the SAM template correctly in my Lambda so that I can refer the existing bucket properly instead of creating the new one.

Thank you

Does this answer your question? [AWS SAM YAML File not able to refer to an existing bucket for S3 event](https://stackoverflow.com/questions/60066931/aws-sam-yaml-file-not-able-to-refer-to-an-existing-bucket-for-s3-event) — Pat Myron, Sep 13 '20 at 21:17

score 1 · Answer 1 · answered Sep 13 '20 at 11:30

1

Any items listed under the Resources section refer to the resources the stack is responsible for maintaining.

When you list SrcBucket you are asking for CloudFormation to create a new S3 bucket with the name being the value of !ImportValue global-bucket which is the name of an S3 bucket you have already created.

Assuming that this is the bucket name you can simply reference it in your template as shown below.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  hellolambda

  Sample SAM Template for hellolambda
  
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Events:
        HelloLambdaEvent:
         Type: S3
         Properties:
            Bucket: !ImportValue global-bucket
            Events: s3:ObjectCreated:*

answered Sep 13 '20 at 11:30

Chris Williams

32,215
4
30
68

With this, I'm getting this error:- Waiting for changeset to be created.. Error: Failed to create changeset for the stack: my-lambda-stack, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [HelloWorldFunction] is invalid. Event with id [HelloLambdaEvent] is invalid. S3 events must reference an S3 bucket in the same template. – vinod827 Sep 13 '20 at 11:38
Unfortunately if the limitation is it must be in the same bucket you would need to use either https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html (replacing the serverless function resource) or have a shared template with the bucket and all Lambda functions – Chris Williams Sep 13 '20 at 11:41
sorry Chris, did not understand...can you give some reference pointers...for me to check...how can I use the events like the way we have in serverless lambda in the normal lambda function? – vinod827 Sep 13 '20 at 11:58
Right, so this be a bit tricky. The S3 event itself is attached to the S3 resource which means that as you want to reuse the S3 bucket itself you would actually need to follow a different route on this. If you're not specifically tied to the S3 events, you can use CloudWatch Events instead. Take a look at [this tutorial](https://docs.aws.amazon.com/codepipeline/latest/userguide/create-cloudtrail-S3-source-cfn.html). You would set the target as a Lambda – Chris Williams Sep 13 '20 at 12:30
The documentation here includes an example of the target as a Lambda: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html – Chris Williams Sep 13 '20 at 12:31

Instead of referring an existing AWS S3 bucket, Cloud Formation is trying to create the bucket

1 Answers1