16

I'm trying to modify this AWS-provided CDK example to instead use an existing bucket. Additional documentation indicates that importing existing resources is supported. So far I am unable to add an event notification to the existing bucket using CDK.

Here is my modified version of the example:

class S3TriggerStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # create lambda function
        function = _lambda.Function(self, "lambda_function",
                                    runtime=_lambda.Runtime.PYTHON_3_7,
                                    handler="lambda-handler.main",
                                    code=_lambda.Code.asset("./lambda"))

        # **MODIFIED TO GET EXISTING BUCKET**
        #s3 = _s3.Bucket(self, "s3bucket")
        s3 = _s3.Bucket.from_bucket_arn(self, 's3_bucket',
            bucket_arn='arn:<my_region>:::<my_bucket>')

        # create s3 notification for lambda function
        notification = aws_s3_notifications.LambdaDestination(function)

        # assign notification for the s3 event type (ex: OBJECT_CREATED)
        s3.add_event_notification(_s3.EventType.OBJECT_CREATED, notification)

This results in the following error when trying to add_event_notification:

AttributeError: '_IBucketProxy' object has no attribute 'add_event_notification'

The from_bucket_arn function returns an IBucket, and the add_event_notification function is a method of the Bucket class, but I can't seem to find any other way to do this. Maybe it's not supported. Any help would be appreciated.

cyber-samurai
  • 161
  • 1
  • 1
  • 4

11 Answers11

12

since June 2021 there is a nicer way to solve this problem. Since approx. Version 1.110.0 of the CDK it is possible to use the S3 notifications with Typescript Code:

Example:

const s3Bucket = s3.Bucket.fromBucketName(this, 'bucketId', 'bucketName');
s3Bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaFunction), {
    prefix: 'example/file.txt'
});

CDK Documentation: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-notifications-readme.html

Pull Request: https://github.com/aws/aws-cdk/pull/15158

Kilian Pfeifer
  • 191
  • 2
  • 5
11

I managed to get this working with a custom resource. It's TypeScript, but it should be easily translated to Python:

const uploadBucket = s3.Bucket.fromBucketName(this, 'BucketByName', 'existing-bucket');

const fn = new lambda.Function(this, 'MyFunction', {
    runtime: lambda.Runtime.NODEJS_10_X,
    handler: 'index.handler',
    code: lambda.Code.fromAsset(path.join(__dirname, 'lambda-handler'))
});

const rsrc = new AwsCustomResource(this, 'S3NotificationResource', {
    onCreate: {
        service: 'S3',
        action: 'putBucketNotificationConfiguration',
        parameters: {
            // This bucket must be in the same region you are deploying to
            Bucket: uploadBucket.bucketName,
            NotificationConfiguration: {
                LambdaFunctionConfigurations: [
                    {
                        Events: ['s3:ObjectCreated:*'],
                        LambdaFunctionArn: fn.functionArn,
                        Filter: {
                            Key: {
                                FilterRules: [{ Name: 'suffix', Value: 'csv' }]
                            }
                        }
                    }
                ]
            }
        },
        // Always update physical ID so function gets executed
        physicalResourceId: 'S3NotifCustomResource' + Date.now().toString()
    }
});

fn.addPermission('AllowS3Invocation', {
    action: 'lambda:InvokeFunction',
    principal: new iam.ServicePrincipal('s3.amazonaws.com'),
    sourceArn: uploadBucket.bucketArn
});

rsrc.node.addDependency(fn.permissionsNode.findChild('AllowS3Invocation'));

This is basically a CDK version of the CloudFormation template laid out in this example. See the docs on the AWS SDK for the possible NotificationConfiguration parameters.

James Irwin
  • 1,171
  • 8
  • 21
  • 1
    It does not worked for me. Error says: Access Denied – KpsLok Jan 29 '20 at 02:36
  • It doesn't work for me, neither. @user400483's answer works for me. My cdk version is 1.62.0 (build 8c2d7fc) – Kazuya Tomita Sep 14 '20 at 03:34
  • 1
    I have to change the `physicalResourceId` to `cr.PhysicalResourceId.of("S3NotificationResource" + Date.now().toString())` – xwlee Nov 03 '20 at 02:52
  • Also take note that the argument of `s3.Bucket.fromBucketName` is scope, id, bucket name. If not you will be access denied because passing the id as bucket name – xwlee Nov 03 '20 at 02:54
  • Even today, a simpler way to add a S3 notification to an existing S3 bucket still on its road, https://github.com/aws/aws-cdk/pull/11773. Since my scenario is a Lambda function plus S3 event, I voted up this answer. Thanks so much! – Scott Hsieh Jun 05 '21 at 14:51
8

Sorry I can't comment on the excellent James Irwin's answer above due to a low reputation, but I took and made it into a Construct.

The comment about "Access Denied" took me some time to figure out too, but the crux of it is that the function is S3:putBucketNotificationConfiguration, but the IAM Policy action to allow is S3:PutBucketNotification.

Here's the [code for the construct]:(https://gist.github.com/archisgore/0f098ae1d7d19fddc13d2f5a68f606ab)

import * as cr from '@aws-cdk/custom-resources';
import * as logs from '@aws-cdk/aws-logs';
import * as s3 from '@aws-cdk/aws-s3';
import * as sqs from '@aws-cdk/aws-sqs';
import * as iam from '@aws-cdk/aws-iam';
import {Construct} from '@aws-cdk/core';

// You can drop this construct anywhere, and in your stack, invoke it like this:
// const s3ToSQSNotification = new S3NotificationToSQSCustomResource(this, 's3ToSQSNotification', existingBucket, queue);

export class S3NotificationToSQSCustomResource extends Construct {

    constructor(scope: Construct, id: string, bucket: s3.IBucket, queue: sqs.Queue) {
        super(scope, id);

        // https://stackoverflow.com/questions/58087772/aws-cdk-how-to-add-an-event-notification-to-an-existing-s3-bucket
        const notificationResource = new cr.AwsCustomResource(scope, id+"CustomResource", {
            onCreate: {
                service: 'S3',
                action: 'putBucketNotificationConfiguration',
                parameters: {
                    // This bucket must be in the same region you are deploying to
                    Bucket: bucket.bucketName,
                    NotificationConfiguration: {
                        QueueConfigurations: [
                            {
                                Events: ['s3:ObjectCreated:*'],
                                QueueArn: queue.queueArn,
                            }
                        ]
                    }
                },
                physicalResourceId: <cr.PhysicalResourceId>(id + Date.now().toString()),
            },
            onDelete: {
                service: 'S3',
                action: 'putBucketNotificationConfiguration',
                parameters: {
                    // This bucket must be in the same region you are deploying to
                    Bucket: bucket.bucketName,
                    // deleting a notification configuration involves setting it to empty.
                    NotificationConfiguration: {
                    }
                },
                physicalResourceId: <cr.PhysicalResourceId>(id + Date.now().toString()),
            },
            policy: cr.AwsCustomResourcePolicy.fromStatements([new iam.PolicyStatement({
                // The actual function is PutBucketNotificationConfiguration.
                // The "Action" for IAM policies is PutBucketNotification.
                // https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html#amazons3-actions-as-permissions
                actions: ["S3:PutBucketNotification"],
                 // allow this custom resource to modify this bucket
                resources: [bucket.bucketArn],
            })]),
            logRetention: logs.RetentionDays.ONE_DAY,
        });

        // allow S3 to send notifications to our queue
        // https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#grant-destinations-permissions-to-s3
        queue.addToResourcePolicy(new iam.PolicyStatement({
            principals: [new iam.ServicePrincipal("s3.amazonaws.com")],
            actions: ["SQS:SendMessage"],
            resources: [queue.queueArn],
            conditions: {
                ArnEquals: {"aws:SourceArn": bucket.bucketArn}
            }
        }));

        // don't create the notification custom-resource until after both the bucket and queue
        // are fully created and policies applied.
        notificationResource.node.addDependency(bucket);
        notificationResource.node.addDependency(queue);
    }
}
Ron U
  • 498
  • 1
  • 5
  • 21
user400483
  • 116
  • 1
  • 2
  • the custom resource will overwrite any existing notification from the bucket, how can you overcome it? (e.g. when you want to add notifications for multiple resources) – Ron U Jan 13 '21 at 09:24
6

UPDATED: Source code from original answer will overwrite existing notification list for bucket which will make it impossible adding new lambda triggers. Here's the solution which uses event sources to handle mentioned problem.

import aws_cdk {
    aws_s3 as s3,
    aws_cdk.aws_lambda as lambda_
    aws_lambda_event_sources as event_src
}
import path as path

class S3LambdaTrigger(core.Stack):
    
    def __init__(self, scope: core.Construct, id: str):
        
        super().__init__(scope, id)
        
        bucket = s3.Bucket(
            self, "S3Bucket",
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            bucket_name='BucketName',
            encryption=s3.BucketEncryption.S3_MANAGED,
            versioned=True
        )

        fn = lambda_.Function(
            self, "LambdaFunction",
            runtime=lambda_.Runtime.NODEJS_10_X,
            handler="index.handler",
            code=lambda_.Code.from_asset(path.join(__dirname, "lambda-handler"))
        )

        fn.add_permission(
            's3-service-principal', 
            principal=aws_iam.ServicePrincipal('s3.amazonaws.com')
        )

        fn.add_event_source(
            event_src.S3EventSource(
                bucket, 
                events=[s3.EventType.OBJECT_CREATED, s3.EventType.OBJECT_REMOVED],
                filters=[s3.NotificationKeyFilter(prefix="subdir/", suffix=".txt")]
            )
        )

ORIGINAL: I took ubi's solution in TypeScript and successfully translated it to Python. His solution worked for me.

#!/usr/bin/env python

from typing import List

from aws_cdk import (
    core,
    custom_resources as cr,
    aws_lambda as lambda_,
    aws_s3 as s3,
    aws_iam as iam,
)


class S3NotificationLambdaProps:
    def __init__(self, bucket: s3.Bucket, function: lambda_.Function, events: List[str], prefix: str):
        self.bucket = bucket
        self.function = function
        self.events = events
        self.prefix = prefix


class S3NotificationLambda(core.Construct):

    def __init__(self, scope: core.Construct, id: str, props: S3NotificationLambdaProps):
        super().__init__(scope, id)

        self.notificationResource = cr.AwsCustomResource(
            self, f'CustomResource{id}',
            on_create=cr.AwsSdkCall(
                service="S3",
                action="S3:putBucketNotificationConfiguration",
                # Always update physical ID so function gets executed
                physical_resource_id=cr.PhysicalResourceId.of(f'S3NotifCustomResource{id}'),
                parameters={
                    "Bucket": props.bucket.bucket_name,
                    "NotificationConfiguration": {
                        "LambdaFunctionConfigurations": [{
                            "Events": props.events,
                            "LambdaFunctionArn": props.function.function_arn,
                            "Filter": {
                                "Key": {"FilterRules": [{"Name": "prefix", "Value": props.prefix}]}
                            }}
                        ]
                    }
                }
            ),
            on_delete=cr.AwsSdkCall(
                service="S3",
                action="S3:putBucketNotificationConfiguration",
                # Always update physical ID so function gets executed
                physical_resource_id=cr.PhysicalResourceId.of(f'S3NotifCustomResource{id}'),
                parameters={
                    "Bucket": props.bucket.bucket_name,
                    "NotificationConfiguration": {},
                }
            ),
            policy=cr.AwsCustomResourcePolicy.from_statements(
                statements=[
                    iam.PolicyStatement(
                        actions=["S3:PutBucketNotification", "S3:GetBucketNotification"],
                        resources=[props.bucket.bucket_arn]
                    ),
                ]
            )
        )

        props.function.add_permission(
            "AllowS3Invocation",
            action="lambda:InvokeFunction",
            principal=iam.ServicePrincipal("s3.amazonaws.com"),
            source_arn=props.bucket.bucket_arn,
        )

        # don't create the notification custom-resource until after both the bucket and lambda
        # are fully created and policies applied.
        self.notificationResource.node.add_dependency(props.bucket)
        self.notificationResource.node.add_dependency(props.function)
# Usage:

s3NotificationLambdaProps = S3NotificationLambdaProps(
    bucket=bucket_,
    function=lambda_fn_,
    events=['s3:ObjectCreated:*'],
    prefix='foo/'
)

s3NotificationLambda = S3NotificationLambda(
    self, "S3NotifLambda",
    self.s3NotificationLambdaProps
)
  • 1
    This is working only when one trigger is implemented on a bucket. But when I have more than one trigger on the same bucket, due to the use of 'putBucketNotificationConfiguration' it is replacing the existing configuration. – BraveNinja Sep 21 '20 at 22:15
  • Thank you @BraveNinja! I updated my answer with other solution. – Yerzhan Bissaliyev Sep 22 '20 at 04:22
  • 2
    your updated code uses a new bucket rather than an existing bucket -- the original question is about setting up these notifications on an existing bucket (IBucket rather than Bucket) – alex9311 Sep 25 '20 at 00:13
  • @alex9311 you can import existing bucket with the following code `bucket = s3.Bucket.from_bucket_name(self, "S3Bucket", "BucketName")` instead of creating new bucket and apply rest of the code for attaching event trigger – Yerzhan Bissaliyev Sep 25 '20 at 06:22
  • 3
    unfortunately that doesn't work, once you use `from_bucket_name` you can no longer pass the bucket to S3EventSource. See https://github.com/aws/aws-cdk/issues/5364 – alex9311 Oct 01 '20 at 19:03
3

Here is a python solution for adding / replacing a lambda trigger to an existing bucket including the filter. @James Irwin your example was very helpful.
Thanks to @JørgenFrøland for pointing out that the custom resource config will replace any existing notification triggers based on the boto3 documentation https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.BucketNotification.put

One note is he access denied issue is because if you do putBucketNotificationConfiguration action the policy creates a s3:PutBucketNotificationConfiguration action but that action doesn't exist https://github.com/aws/aws-cdk/issues/3318#issuecomment-584737465 Same issue happens if you set the policy using AwsCustomResourcePolicy.fromSdkCalls I've added a custom policy that might need to be restricted further.

s3_bucket = s3.Bucket.from_bucket_name(
    self, 's3-bucket-by-name', 'existing-bucket-name')

trigger_lambda = _lambda.Function(
    self,
    '{id}-s3-trigger-lambda',
    environment=lambda_env,
    code=_lambda.Code.from_asset('./ladle-sink/'),
    runtime=_lambda.Runtime.PYTHON_3_7,
    handler='lambda_function.lambda_handler',
    memory_size=512,
    timeout=core.Duration.minutes(3))

trigger_lambda.add_permission(
    's3-trigger-lambda-s3-invoke-function',
    principal=iam.ServicePrincipal('s3.amazonaws.com'),
    action='lambda:InvokeFunction',
    source_arn=base_resources.incoming_documents_bucket.bucket_arn)

custom_s3_resource = _custom_resources.AwsCustomResource(
    self,
    's3-incoming-documents-notification-resource',
    policy=_custom_resources.AwsCustomResourcePolicy.from_statements([
        iam.PolicyStatement(
            effect=iam.Effect.ALLOW,
            resources=['*'],
            actions=['s3:PutBucketNotification']
        )
    ]),
    on_create=_custom_resources.AwsSdkCall(
        service="S3",
        action="putBucketNotificationConfiguration",
        parameters={
            "Bucket": s3_bucket.bucket_name,
            "NotificationConfiguration": {
                "LambdaFunctionConfigurations": [
                    {
                        "Events": ['s3:ObjectCreated:*'],
                        "LambdaFunctionArn": trigger_lambda.function_arn,
                        "Filter": {
                            "Key": {
                                "FilterRules": [
                                    {'Name': 'suffix', 'Value': 'html'}]
                            }
                        }
                    }
                ]
            }
        },
        physical_resource_id=_custom_resources.PhysicalResourceId.of(
            f's3-notification-resource-{str(uuid.uuid1())}'),
        region=env.region
    ))

custom_s3_resource.node.add_dependency(
    trigger_lambda.permissions_node.find_child(
        's3-trigger-lambda-s3-invoke-function'))
alex9311
  • 1,230
  • 1
  • 18
  • 42
Yan
  • 3,533
  • 4
  • 24
  • 45
  • Will this overwrite the entire list of notifications on the bucket or append if there are already notifications connected to the bucket?The reason I ask is that this doc: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.BucketNotification.put says: "Using this API, you can replace an existing notification configuration." And I suspect it uses the same SDK-call. – Jørgen Frøland Aug 06 '20 at 10:48
  • @JørgenFrøland From documentation it looks like it will replace the existing triggers and you would have to configure all the triggers in this custom resource. I will update the answer that it replaces. It is part of the CDK deploy which creates the S3 bucket and it make sense to add all the triggers as part of the custom resource. Thanks! – Yan Aug 06 '20 at 11:34
  • In that case, it may be a better solution to add a post-deploy job that uses boto3 (for Python) and add the new notification. Then it's possible to load the existing notifications, append the new, and finally write back. The only problem may be to get access to the correct ARNs. – Jørgen Frøland Aug 06 '20 at 13:33
  • @JørgenFrøland That depends on the use case. I am using AWS CDK for deploying / destroying all of infra. I want to have all the config in the stack. I don't see a reason to have post-deploy scripts. For some people who is adding / updating parts of infra might be useful. – Yan Aug 06 '20 at 13:43
  • 1
    I just figured that its quite easy to load the existing config using boto3 and append it to the new config. Then a post-deploy-script should not be necessary after all. – Jørgen Frøland Aug 06 '20 at 13:50
  • 1
    I also experience that the notification config remains on the bucket after destroying the stack. Anyone experiencing the same? In that case, an "on_delete" parameter is useful to clean up. – Jørgen Frøland Aug 07 '20 at 08:35
  • 1
    I had to add an on_update (well, onUpdate, because I'm doing Typescript) parameter as well. – Aurelia Peters Aug 27 '20 at 21:51
  • this code works for me (if I add on_update code with the same as on_create and use a static value for the id instead of uuid1). The problem I run into is when destroying the stack, the event fails to delete and I have to go delete it manually – alex9311 Oct 01 '20 at 20:54
  • @alex9311what is the error when you try to destroy the stack? Does this help https://aws.amazon.com/premiumsupport/knowledge-center/best-practices-custom-cf-lambda/ – Yan Oct 07 '20 at 03:45
1

Thanks to the great answers above, see below for a construct for s3 -> lambda notification. It can be used like

    const fn = new SingletonFunction(this, "Function", {
    ...
    });

    const bucket = Bucket.fromBucketName(this, "Bucket", "...");

    const s3notification = new S3NotificationLambda(this, "S3Notification", {
      bucket: bucket,
      lambda: function,
      events: ['s3:ObjectCreated:*'],
      prefix: "some_prefix/"
    })

Construct (drop-in to your project as a .ts file)

import * as cr from "@aws-cdk/custom-resources";
import * as logs from "@aws-cdk/aws-logs";
import * as s3 from "@aws-cdk/aws-s3";
import * as sqs from "@aws-cdk/aws-sqs";
import * as iam from "@aws-cdk/aws-iam";
import { Construct } from "@aws-cdk/core";
import * as lambda from "@aws-cdk/aws-lambda";

export interface S3NotificationLambdaProps {
  bucket: s3.IBucket;
  lambda: lambda.IFunction;
  events: string[];
  prefix: string;
}

export class S3NotificationLambda extends Construct {
  constructor(scope: Construct, id: string, props: S3NotificationLambdaProps) {
    super(scope, id);

    const notificationResource = new cr.AwsCustomResource(
      scope,
      id + "CustomResource",
      {
        onCreate: {
          service: "S3",
          action: "putBucketNotificationConfiguration",
          parameters: {
            // This bucket must be in the same region you are deploying to
            Bucket: props.bucket.bucketName,
            NotificationConfiguration: {
              LambdaFunctionConfigurations: [
                {
                  Events: props.events,
                  LambdaFunctionArn: props.lambda.functionArn,
                  Filter: {
                    Key: {
                      FilterRules: [{ Name: "prefix", Value: props.prefix }],
                    },
                  },
                },
              ],
            },
          },
          physicalResourceId: <cr.PhysicalResourceId>(
            (id + Date.now().toString())
          ),
        },
        onDelete: {
          service: "S3",
          action: "putBucketNotificationConfiguration",
          parameters: {
            // This bucket must be in the same region you are deploying to
            Bucket: props.bucket.bucketName,
            // deleting a notification configuration involves setting it to empty.
            NotificationConfiguration: {},
          },
          physicalResourceId: <cr.PhysicalResourceId>(
            (id + Date.now().toString())
          ),
        },
        policy: cr.AwsCustomResourcePolicy.fromStatements([
          new iam.PolicyStatement({
            // The actual function is PutBucketNotificationConfiguration.
            // The "Action" for IAM policies is PutBucketNotification.
            // https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html#amazons3-actions-as-permissions
            actions: ["S3:PutBucketNotification", "S3:GetBucketNotification"],
            // allow this custom resource to modify this bucket
            resources: [props.bucket.bucketArn],
          }),
        ]),
      }
    );

    props.lambda.addPermission("AllowS3Invocation", {
      action: "lambda:InvokeFunction",
      principal: new iam.ServicePrincipal("s3.amazonaws.com"),
      sourceArn: props.bucket.bucketArn,
    });

    // don't create the notification custom-resource until after both the bucket and queue
    // are fully created and policies applied.
    notificationResource.node.addDependency(props.bucket);
    notificationResource.node.addDependency(props.lambda);
  }
}

ubi
  • 4,041
  • 3
  • 33
  • 50
1

based on the answer from @ubi

in case of you don't need the SingletonFunction but Function + some cleanup

call like this:

const s3NotificationLambdaProps = < S3NotificationLambdaProps > {
    bucket: bucket,
    lambda: lambda,
    events: ['s3:ObjectCreated:*'],
    prefix: '', // or put some prefix
};

const s3NotificationLambda = new S3NotificationLambda(this, `${envNameUpperCase}S3ToLambdaNotification`, s3NotificationLambdaProps);

and the construct will be like this:

import * as cr from "@aws-cdk/custom-resources";
import * as s3 from "@aws-cdk/aws-s3";
import * as iam from "@aws-cdk/aws-iam";
import { Construct } from "@aws-cdk/core";
import * as lambda from "@aws-cdk/aws-lambda";

export interface S3NotificationLambdaProps {
    bucket: s3.IBucket;
    lambda: lambda.Function;
    events: string[];
    prefix: string;
}

export class S3NotificationLambda extends Construct {
    constructor(scope: Construct, id: string, props: S3NotificationLambdaProps) {
        super(scope, id);

        const notificationResource = new cr.AwsCustomResource(
            scope,
            id + "CustomResource", {
                onCreate: {
                    service: "S3",
                    action: "putBucketNotificationConfiguration",
                    parameters: {
                        // This bucket must be in the same region you are deploying to
                        Bucket: props.bucket.bucketName,
                        NotificationConfiguration: {
                            LambdaFunctionConfigurations: [{
                                Events: props.events,
                                LambdaFunctionArn: props.lambda.functionArn,
                                Filter: {
                                    Key: {
                                        FilterRules: [{
                                            Name: "prefix",
                                            Value: props.prefix
                                        }],
                                    },
                                },
                            }, ],
                        },
                    },
                    physicalResourceId: < cr.PhysicalResourceId > (
                        (id + Date.now().toString())
                    ),
                },
                onDelete: {
                    service: "S3",
                    action: "putBucketNotificationConfiguration",
                    parameters: {
                        // This bucket must be in the same region you are deploying to
                        Bucket: props.bucket.bucketName,
                        // deleting a notification configuration involves setting it to empty.
                        NotificationConfiguration: {},
                    },
                    physicalResourceId: < cr.PhysicalResourceId > (
                        (id + Date.now().toString())
                    ),
                },
                policy: cr.AwsCustomResourcePolicy.fromStatements([
                    new iam.PolicyStatement({
                        // The actual function is PutBucketNotificationConfiguration.
                        // The "Action" for IAM policies is PutBucketNotification.
                        // https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html#amazons3-actions-as-permissions
                        actions: ["S3:PutBucketNotification", "S3:GetBucketNotification"],
                        // allow this custom resource to modify this bucket
                        resources: [props.bucket.bucketArn],
                    }),
                ]),
            }
        );

        props.lambda.addPermission("AllowS3Invocation", {
            action: "lambda:InvokeFunction",
            principal: new iam.ServicePrincipal("s3.amazonaws.com"),
            sourceArn: props.bucket.bucketArn,
        });

        // don't create the notification custom-resource until after both the bucket and lambda
        // are fully created and policies applied.
        notificationResource.node.addDependency(props.bucket);
        notificationResource.node.addDependency(props.lambda);
    }
}
Aurelia Peters
  • 2,169
  • 1
  • 20
  • 34
Vitalii
  • 23
  • 6
  • This seems to remove existing notifications, which means that I can't have many lambdas listening on an existing bucket. any ideas? – NiRR Sep 13 '20 at 10:24
  • @NiRR you could use a fan-out lambda to distribute your events, unfortunately I faced the same limitation about having the only one lambda per bucket notification. So far I haven't found any other solution regarding this. – Vitalii Sep 14 '20 at 13:14
1

This is CDK solution.

  1. Get a grab of existing bucket using fromBucketAttributes
  2. Then for your bucket, use addEventNotification to trigger your lambda.
declare const myLambda: lambda.Function;
const bucket = s3.Bucket.fromBucketAttributes(this, 'ImportedBucket', {
  bucketArn: 'arn:aws:s3:::my-bucket',
});

// now you can just call methods on the bucket
bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(myLambda), {prefix: 'home/myusername/*'});

More details can be found here

codebusta
  • 1,922
  • 1
  • 14
  • 15
  • This is almost perfect, however the prefix does not support wildcards – tiktock Jun 18 '23 at 07:49
  • @tiktock, you must be right, i think there is no need for wildcard in this case. I think the [CDK 1 documentation](https://docs.aws.amazon.com/cdk/api/v1/docs/aws-s3-readme.html#importing-existing-buckets) got it wrong as well. – codebusta Jun 26 '23 at 08:27
1

AWS now supports s3 eventbridge events, which allows for adding a source s3 bucket by name. So this worked for me. Note that you need to enable eventbridge events manually for the triggering s3 bucket.

  new Rule(this, 's3rule', {
                   eventPattern: {
                       source: ['aws.s3'],
                       detail: {
                           'bucket': {'name': ['existing-bucket']},
                           'object': {'key' : [{'prefix' : 'prefix'}]}
                       },
                       detailType: ['Object Created']
                   },
                   targets: [new targets.LambdaFunction(MyFunction)]
               }
           );
TomB
  • 51
  • 4
1

With the newer functionality, in python this can now be done as:

bucket = aws_s3.Bucket.from_bucket_name(
    self, "bucket", "bucket-name"
)


bucket.add_event_notification(
    aws_s3.EventType.OBJECT_CREATED,
    aws_s3_notifications.LambdaDestination(your_lambda),
    aws_s3.NotificationKeyFilter(
        prefix="prefix/path/",
    ),
)

At the time of writing, the AWS documentation seems to have the prefix arguments incorrect in their examples so this was moderately confusing to figure out.

Thanks to @Kilian Pfeifer for starting me down the right path with the typescript example.

Scott Brenstuhl
  • 627
  • 6
  • 7
0

I used CloudTrail for resolving the issue, code looks like below and its more abstract:

const trail = new cloudtrail.Trail(this, 'MyAmazingCloudTrail');

const options: AddEventSelectorOptions = {
  readWriteType: cloudtrail.ReadWriteType.WRITE_ONLY
};

// Adds an event selector to the bucket 
trail.addS3EventSelector([{
  bucket: bucket, // 'Bucket' is of type s3.IBucket,
}], options);

bucket.onCloudTrailWriteObject('MyAmazingCloudTrail', {
  target: new targets.LambdaFunction(functionReference)
});
Kanika
  • 1
  • 1