How to modify AWS EventBridge Rule to use AND instead of OR filter logic?

Question

I want to trigger an AWS lambda function via EventBridge every time an S3 Object is created in an S3 bucket called "mybucket", but ONLY if its name/key ends with a ".csv"-suffix AND if it was created within the "in"-folder of that bucket. The EventBridge Rule that I currently have is this:

{
  "detail-type": ["Object Created"],
  "source": ["aws.s3"],
  "detail": {
    "bucket": {
      "name": ["mybucket"]
    },
    "object": {
      "key": [{
        "suffix": ".csv"
      }, {
        "prefix": "in/"
      }]
    }
  }
}

I would actually expect this rule to work the correct way BUT it is not, instead it behaves as if there was an OR relation between the suffix and prefix filter conditions. As I understand the AWS Documentation (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns-content-based-filtering.html#eb-filtering-complex-example) the above rule should define an AND relation between the suffix and prefix filter conditions similar to this example given in the documentation:

{
  "time": [ { "prefix": "2017-10-02" } ],
  "detail": {
    "state": [ { "anything-but": "initializing" } ],
    "c-count": [ { "numeric": [ ">", 0, "<=", 5 ] } ],
    "d-count": [ { "numeric": [ "<", 10 ] } ],
    "x-limit": [ { "anything-but": [ 100, 200, 300 ] } ]
  }
}

Whereas an OR relation would require an extra $or-syntax as in this example given in the documentation:

{
  "detail": {
    "$or": [
      { "c-count": [ { "numeric": [ ">", 0, "<=", 5 ] } ] },
      { "d-count": [ { "numeric": [ "<", 10 ] } ] },
      { "x-limit": [ { "numeric": [ "=", 3.018e2 ] } ] }
    ]
  }
}

So, why is my rule behaving as if there was an OR relation between the suffix and prefix conditions? And what do I need to change to get it working the way I want?

S3 object key starts with leading slash, try changing `in/` to `/in/`. — Ankush Jain, Dec 15 '22 at 12:38
@AnkushJain S3 keys and prefixes don't typically start with forward slash. — jarmod, Dec 15 '22 at 13:37
It is as @jarmod stated. The s3 object keys are not starting with a forward slash. I can actually see the object keys in the logged events, and they look like: "in/ipsumlorem...". My problem is not that the prefix or suffix filter rules of the "Object Created" events are not working at all but that they are not working correctly in combination. I'm getting an OR behavior where I would expect an AND behavior. — jimmyorpheus, Dec 15 '22 at 13:53
@jimmyorpheus Can you please try `[{ "suffix": ".csv" , "prefix": "in/" }]` ? One item in an array instead 2 items. — Ankush Jain, Dec 15 '22 at 14:27
@AnkushJain I am using AWS Cloudformation to deploy my infrastructure. I have tried your solution but Cloudformation fails with the following error: Event pattern is not valid. Reason: Only one key allowed in match expression at [Source: (String)"{"detail-type":["Object Created"],"source":["aws.s3"],"detail":{"bucket":{"name":["mybucket"]},"object":{"key":[{"prefix":"in/","suffix":".csv"}]}}}"; line: 1, column: 151] (Service: AmazonCloudWatchEvents; Status Code: 400; Error Code: InvalidEventPatternException; Request ID: ...; Proxy: null) — jimmyorpheus, Dec 15 '22 at 15:31

score 1 · Accepted Answer · answered May 23 '23 at 14:21

This is not possible at the moment

There are two ways to setup something that "look like and operator` and do not throw syntax errors, but they will work differently:

Two keys with different filter (as proposed by Peter Bouwdewijn) - the latter filter will overwrite the former, so it will only filter by suffix, prefix will be completely ignored:
```
"key": [{"prefix": "example/directory/"}],
"key": [{"suffix": ".png"}]
```
Input "key": "other/directory/image.png" will be valid
Provide two filter objects in the same array - they will act as OR operator:
```
"key": [{"prefix": "example/directory/"}, {"suffix": ".png"}]
```
Both inputs "key": "other/directory/image.png" and "key": "example/directory/not_image.txt" will be valid

See Content-based filtering and Arrays pages of the AWS EventBridge documentation for more info

score -1 · Answer 2 · answered Apr 24 '23 at 11:27

Sounds like the exact problem I am facing. I found something in IBM docs: https://www.ibm.com/docs/en/dsm?topic=csqcson-forwarding-objectcreated-notifications-sqs-queue-by-using-amazon-eventbridge

There they state to repeat the key

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["<example-bucket>"]
    },
    "object": {
      "key": [{
        "prefix": "example/directory/"
      }],
      "key": [{
        "suffix": ".png"
      }]
    }
  }
}

This is very counter intuitive and even goes against the docs from AWS. I have not tried it yet.

Any way to filter out suffix in a anything-but? "SourceIdentifier": [{ "anything-but": { "suffix": "-test" } }] — John Pham, Jul 22 '23 at 23:28

How to modify AWS EventBridge Rule to use AND instead of OR filter logic?

2 Answers2