Can I trigger an ECS/Fargate task from a specific file upload in S3?

Question

I know that I can trigger a task when a file is uploaded (per https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatch-Events-tutorial-ECS.html) however, how can I trigger a task when a specific file is uploaded?

Amazon seems not to have anticipated people having multiple jobs watching the same bucket for different files :(

@MunavirChavody S3 will generate an event when any file lands in a bucket. I want to see an event only when foo/bar/myfile.* lands in the bucket — Steve, Aug 27 '19 at 23:03

score 7 · Accepted Answer · answered Aug 28 '19 at 01:19

You can accomplish this with CloudWatch Events from CloudTrail Data Events.

Head over to CloudTrail, and create a Trail for your account.

For Apply trail to all regions, choose No.
Under Management events, Read/Write Events, select none.
Under Data events, select S3. Input your S3 bucket name and folder name (prefix) to log data events for, and select Write (don't set read).
Under Storage location, create a new bucket or provide a bucket to be used to store the log files.
Select Create

Next, create a CloudWatch Event rule that targets your ECS Task when the CloudTrail Data Event happens.

Head over to CloudWatch and Create a new Event rule.

For the Event Source select Event Pattern
Change the dropdown that says "Build event pattern to match events by service" to select "Custom Event Pattern"
Enter the event pattern below:

{
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject"
    ],
    "requestParameters": {
      "bucketName": [
        "your-bucket-name" // this is the bucket where your events are happening
      ],
      "key": [
        "your-object-key" // this is the object key you want to trigger starting your ECS task, note that it's an array.
      ]
    }
  }
}

Customize the bucketName and key above as appropriate for your use.
For your target, select ECS Task, configure the task as appropriate.
Select Configure details, give the rule a name and set the State to Enabled, and click Create rule.

Now that your rule is enabled, when you upload an object with the specified key to the specified bucket, CloudWatch Events will trigger the ECS Task you specified.

great answer but I wanted to know if it possible to trigger different ECS tasks for different file names? i.e. — user6248190, Jun 23 '21 at 09:37

score 3 · Answer 2 · answered Jul 16 '20 at 20:36

Looks like you have a wildcard in your comment. To add to the event pattern from hephalump, you can indicate a prefix for the key as well, which will match to any key with that prefix, not just a specific key:

{
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject"
    ],
    "requestParameters": {
      "bucketName": [
        "your-bucket-name" // this is the bucket where your events are happening
      ],
      "key": [
        {"prefix": "path/to/key"}
      ]
    }
  }
}

score 1 · Answer 3 · answered Aug 27 '19 at 22:23

S3 does not generate events for a particular object, rather for events that occur within a bucket. The image below is the S3 console for events that a bucket can send.

Note that there you could use the prefix and/or suffix to perhaps get close to what you want but individual objects cannot generate events. A good option though would be a Lambda that can filter on the object name and, if it matches what you want, run the ECS task.

Can I trigger an ECS/Fargate task from a specific file upload in S3?

3 Answers3

Linked