How to filter an s3 data event by object key suffix on AWS EventBridge

Question

I've created a rule on AWS EventBridge that trigger a Sagemaker Pipeline execution. To do so, I have the following event pattern:

{
  "source": ["aws.s3"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["s3.amazonaws.com"],
    "eventName": ["PutObject", "CopyObject", "CompleteMultipartUpload"],
    "requestParameters": {
      "bucketName": ["my-bucket-name"],
      "key": [{
        "prefix": "folder/inside/my/bucket/"
      }]
    }
  }
}

I have enabled CloudTrail to log my S3 Data Events and the rule are triggering my Sagemaker Pipeline execution correctly.

The problem here is: A pipeline execution are being triggered for all put/copy of any object in my prefix. Then, I would like to trigger my pipeline execution only when a specific object is uploaded in the bucket, by I don't know its entire name.

For instance, possible object name I will have is, where this date is builded dynamically:

my-bucket-name/folder/inside/my/bucket/2021-07-28/_SUCESS

I would like to write an event pattern with something like this:

"prefix": "folder/inside/my/bucket/{current_date}/_SUCCESS"

or

"key": [{
  "prefix": "folder/inside/my/bucket/"
}, {
  "suffix": "_SUCCESS"
}]

I think that Event Pattern on AWS do not support suffix filtering. In the documentation, isn't clear the behavior. I have configured a S3 Event Notification using a suffix and sent the filtered notification to a SQS Queue, but now I don't know what to do with this queue in order to invoke my EventBridge rule to trigger a Sagemaker Pipeline execution.

score 2 · Accepted Answer · edited Nov 01 '22 at 09:09

I was looking at a similar functionality.

Unfortunately, based on the docs from AWS, it looks like it only supports the following patterns:

Comparison	Example	Rule syntax
Null	UserID is null	"UserID": [ null ]
Empty	LastName is empty	"LastName": [""]
Equals	Name is "Alice"	"Name": [ "Alice" ]
And	Location is "New York" and Day is "Monday"	"Location": [ "New York" ], "Day": ["Monday"]
Or	PaymentType is "Credit" or "Debit"	"PaymentType": [ "Credit", "Debit"]
Not	Weather is anything but "Raining"	"Weather": [ { "anything-but": [ "Raining" ] } ]
Numeric (equals)	Price is 100	"Price": [ { "numeric": [ "=", 100 ] } ]
Numeric (range)	Price is more than 10, and less than or equal to 20	"Price": [ { "numeric": [ ">", 10, "<=", 20 ] } ]
Exists	ProductName exists	"ProductName": [ { "exists": true } ]
Does not exist	ProductName does not exist	"ProductName": [ { "exists": false } ]
Begins with	Region is in the US	"Region": [ {"prefix": "us-" } ]

This is duplicated to https://stackoverflow.com/questions/68882690/aws-cloudformation-lambda-and-event-rule-lambda-not-getting-triggered-with-s3 — Potter Rafed, Feb 02 '22 at 14:50
Exactly. Until this moment, isn't possible to define a suffix for the key object. — thatayster, Jun 05 '22 at 06:32

How to filter an s3 data event by object key suffix on AWS EventBridge

1 Answers1