Unable to put notification event to trigger CloudFormation Lambda in existing S3 bucket

Question

I'm migrating a "hand built" AWS infrastructure stack to CloudFormation using a SAM template. I've successfully migrated almost all of the resources, but I've hit a roadblock regarding migrating an S3 bucket. Where I'm running into an issue is with a Lambda function that handles S3 bucket events such as object create and object delete.

I researched and tried several options, including attaching the Lambda trigger event to a pre-existing S3 bucket on stack creation, but I learned that this was not possible because the S3 bucket is not part of the CloudFormation stack being built.

Next I tried to create a new bucket in the CloudFormation stack and attach the event object. At first I tried to use existing SAM template shortcuts to create the event between the S3 buckets and the Lambda, but doing so produced a Circular dependency between resources error that turned out to be due to circular permissions required between the bucket and the lambda.

To work around the circular dependency issue I created an auxiliary custom resource Lambda that runs during the CloudFormation stack creation process that uses the S3 API to PUT the notification directly. I attempted to follow the example solution provided in the first post, wherein one PUTS an event into the S3 bucket using a custom resource. Using this apporach my CloudFormation build always fails with the same error message: Unable to validate the following destination configurations. To solve this issue I found this solution on StackOverflow and added AWS::Lambda::Permission and AWS::S3::BucketPolicy resources to my CloudFormation template. Despite my changes the error still occurs.

What am I missing?

Here is a snippet of my template.yaml:

  IngestClip:
    Type: AWS::Serverless::Function
    Properties:
      Role: !GetAtt LambdaExecutionRole.Arn
      Architectures:
        - arm64
      CodeUri: functions/ingestClip/
      VpcConfig:
        SecurityGroupIds:
          - !Ref LambdaSecurityGroup
        SubnetIds:
          - !Ref PrivateSubnet1
          - !Ref PrivateSubnet2

  IngestClipPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref ConfigureLambdaS3Event
      Action: "lambda:InvokeFunction"
      Principal: s3.amazonaws.com
      SourceAccount: !Ref AWS::AccountId
      SourceArn: !GetAtt IngestBucket.Arn

  IngestBucket:
    Type: AWS::S3::Bucket

  IngestBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref IngestBucket
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: 
              - 's3:PutBucketNotification'
            Resource: !Sub "arn:aws:s3:::${IngestBucket}"
            Principal: 
              AWS: !GetAtt LambdaExecutionRole.Arn

  ConfigureLambdaS3Event:
    DependsOn:
      - NATRoute
      - InternetRoute
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs14.x
      Role: !GetAtt LambdaExecutionRole.Arn
      Timeout: 60
      Architectures:
        - arm64
      VpcConfig:
        SecurityGroupIds:
          - !Ref LambdaSecurityGroup
        SubnetIds:
          - !Ref PrivateSubnet1
          - !Ref PrivateSubnet2
      Layers:
        - !Ref NodeDependencies
        - !Ref Common
      Code:
        ZipFile: |
          const aws = require('aws-sdk');
          const s3 = new aws.S3({ apiVersion: '2006-03-01', signatureVersion: 'v4' });

          const { withCustomResourceResponder } = require('utility/customResource.js');
          const { decorate } = require('utility/decorator.js');

          exports.handler = decorate([
            withCustomResourceResponder({
              shortCircuitOnDeleteRequest: true
            })
          ], async (event, context) => {
            let { Bucket, Function, Event, Filters } = event.ResourceProperties;
            let params = {
              Bucket: Bucket,
              NotificationConfiguration: {
                LambdaFunctionConfigurations: [
                  {
                    LambdaFunctionArn: Function,
                    Events: [
                      Event
                    ],
                    Filter: {
                      Key: {
                        FilterRules: JSON.parse(Filters)
                      }
                    }
                  }
                ]
              }
            };
            console.log('params:', JSON.stringify(params, null, 2));
            return await s3.putBucketNotificationConfiguration(params).promise();
          });


  ConfigureIngestClipEventOnBuild:
    DependsOn:
      - ConfigureLambdaS3Event
      - ConfigureLambdaS3EventIngestBucketAccess
      - IngestBucket
      - IngestClip
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !GetAtt ConfigureLambdaS3Event.Arn
      Bucket: !Ref IngestBucket
      Function: !GetAtt IngestClip.Arn
      Event: 's3:ObjectCreated:*'
      Filters: '[{"Name": "suffix", "Value": ".wav" }]'

Answer 1

Just discovered the issue with my solution: I was configuring the AWS::Lambda::Permission improperly. Instead of granting permission to the function backing the custom resource, it should be granting permission to the actual lambda that the S3 trigger is attaching to, like so:

  IngestClipPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref IngestClip
      Action: "lambda:InvokeFunction"
      Principal: s3.amazonaws.com
      SourceAccount: !Ref AWS::AccountId
      SourceArn: !GetAtt IngestBucket.Arn

Making this change solved the issue, and the events were created successfully.