3

I want to enable DynamoDB streams on my lambda using AWS CDK which I am able to do but I also want to enable the filter criteria on lambda

But I am getting this error:

Invalid filter pattern definition. (Service: AWSLambda; Status Code: 400; Error Code: InvalidParameterValueException

This is the event I am getting from DynamoDB streams:

{
    "input": {
        "Records": [
            {
                "eventID": "e92e0072a661a06df0e62e411f",
                "eventName": "INSERT",
                "eventVersion": "1.1",
                "eventSource": "aws:dynamodb",
                "awsRegion": "<region>",
                "dynamodb": {
                    "ApproximateCreationDateTime": 1639500357,
                    "Keys": {
                        "service": {
                            "S": "service"
                        },
                        "key": {
                            "S": "key"
                        }
                    },
                    "NewImage": {
                        "service": {
                            "S": "service"
                        },
                        "channel": {
                            "S": "email"
                        },
                        "key": {
                            "S": "key"
                        }
                    },
                    "SequenceNumber": "711500000000015864417",
                    "SizeBytes": 168,
                    "StreamViewType": "NEW_IMAGE"
                },
                "eventSourceARN": "arn:aws:dynamodb:<region>:<account>:table/table-name/stream/2021-12-14T13:00:29.888"
            }
        ]
    },
    "env": {
        "lambdaContext": {
            "callbackWaitsForEmptyEventLoop": true,
            "functionVersion": "$LATEST",
            "functionName": "functionName",
            "memoryLimitInMB": "128",
            "logGroupName": "/aws/lambda/functionName",
            "logStreamName": "2021/12/14/[$LATEST]028531c7b489b8ec69bace700acc0",
            "invokedFunctionArn": "arn:aws:lambda:<region>:<account>:function:functionName",
            "awsRequestId": "c72e80252-4722-b9f0-a03b7f8b820e"
        },
        "region": "<region-name>"
    }
}

The event source mapping code is:

const mapping = new lambda.CfnEventSourceMapping(this, 'event', {
  functionName: "functionName,
  batchSize: 1,
  bisectBatchOnFunctionError: true,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  eventSourceArn: <stream-arn>,
  filterCriteria: filter,
});

I want to get the eventName to be INSERT and the channel to be email here. What should be the value of the filter criteria? Its not working for me

Phil Dukhov
  • 67,741
  • 15
  • 184
  • 220
Shobhit Arora
  • 33
  • 1
  • 3

1 Answers1

11

<Edit> CDK filter helpers added in v2.42.0

The original workaround is no longer necessary. The CDK now has event-source filters for Lambda, Kinesis and SQS. Pass the filter to the L2 EventSourceMapping construct:

const source: EventSourceMapping = new lambda.EventSourceMapping(this, "EventSourceMapping",{
    target: func,
    eventSourceArn: table.tableStreamArn,
    startingPosition: lambda.StartingPosition.TRIM_HORIZON,
    filters: [
      lambda.FilterCriteria.filter({
        eventName: lambda.FilterRule.isEqual("INSERT"),
        dynamodb: {  NewImage: { channel: { S: lambda.FilterRule.isEqual("email") } },},
      }),
    ],
  }
);

</Edit>

Here's the DynamoDB streams filter Pattern syntax for new records with a channel of email:

`{ \"eventName\": [\"INSERT\"], \"dynamodb\": { \"NewImage\": {\"channel\": { \"S\" : [\"email\"]}} } }`

In other words, the Pattern is a stringified JSON filter rule with escaped quotes. The pattern is applied against each stream record.

Here is the full CDK syntax. The code starts with the usual L2 EventSourceMapping. It then uses escape hatch syntax to set FilterCriteria on the underlying L1 CfnEventSourceMapping:

// start with the L2 type - Note: the OP code starts with a L1 `CfnEventSourceMapping`
const source: EventSourceMapping = new lambda.EventSourceMapping(this, 'EventSourceMapping', {
  target: func,
  eventSourceArn: table.tableStreamArn,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
});

// escape hatch - get a L1 reference
const cfnSouce = source.node.defaultChild as lambda.CfnEventSourceMapping;

cfnSouce.addPropertyOverride('FilterCriteria', {
  Filters: [
    {
      Pattern: `{ \"eventName\": [\"INSERT\"], \"dynamodb\": { \"NewImage\": {\"channel\": { \"S\" : [\"email\"]}} } }`,
    },
  ],
});
fedonev
  • 20,327
  • 2
  • 25
  • 34
  • It is still not working initially it gave me error in cfnSource.addPropertyOverride that cannot access properties of undefined so i changed it to cfnSource?.addPropertyOverride and it i think cfnSource is undefined that is why the resultant cloudformation prepared has no changes. Is it working for you ?? . – Shobhit Arora Dec 16 '21 at 08:12
  • Yes, the answer's code deploys as expected. Your error is probably due to mixing up [L1 (`Cfn*`) and L2 types](https://docs.aws.amazon.com/cdk/v2/guide/cfn_layer.html#cfn_layer_cfn). My answer starts with the L2 `source: EventSourceMapping` and gets a reference to its L1 child node `cfnSouce: CfnEventSourceMapping`. You _start with_ the L1 construct: `mapping: CfnEventSourceMapping`. Both approaches work fine if done properly. I edited the answer to make our different starting points clearer. – fedonev Dec 16 '21 at 09:46
  • To make it dead simple: if your `mapping` is already a `CfnEventSourceMapping`, you can call `mapping.addPropertyOverride` without the `.node.defaultChild` line. – fedonev Dec 16 '21 at 09:47
  • Yes its working thanks a lot – Shobhit Arora Dec 16 '21 at 10:54
  • Instead of using `addPropertyOverride()`, which takes any string and an `any` value, you should use the `filterCriteria` property. For example: `cfnSourceMapping.filterCriteria = { filters: [{ pattern: "" }] }`. This gives you the benefit of property and type checking. – Corey Jan 24 '23 at 21:16