Including an exit code 1 event in CloudWatch using Terraform for ECS

Question

I've been running containers on ECS, and using AWS Cloudwatch events to notify me when my tasks complete. All of the infrastructure has been created using Terraform. However, I'm unable to get the correct syntax in my event pattern so that I am only notified for non-zero exit codes.

The following resource works great, and sends notifications to SNS every time one of my containers exits:

resource "aws_cloudwatch_event_rule" "container-stopped-rule" {
  name        = "container-stopped"
  description = "Notification for containers that exit for any reason. (error)."

  event_pattern = <<PATTERN
{
  "source": [
    "aws.ecs"
    ],
  "detail-type": [
    "ECS Task State Change"
    ],
  "detail": {

    "lastStatus": [
      "STOPPED"
    ],
"stoppedReason" : [
    "Essential container in task exited"
  ]
}
}
PATTERN
}

However, I'm trying to modify the pattern slightly so that I'm only notified when a container exits with an error code- since we get so many notifications, we've started to tune out the emails and sometimes don't notice the email notifications where containers are exiting with errors:

resource "aws_cloudwatch_event_rule" "container-stopped-rule" {
  name        = "container-stopped"
  description = "Notification for containers with exit code of 1 (error)."

  event_pattern = <<PATTERN
{
  "source": [
    "aws.ecs"
    ],
  "detail-type": [
    "ECS Task State Change"
    ],
  "detail": {
    "containers": [
      {
      "exitCode": 1
      }
    ],
    "lastStatus": [
      "STOPPED"
    ],
"stoppedReason" : [
    "Essential container in task exited"
  ]
}
}
PATTERN
}

This triggers the following error when I terraform apply:

aws_cloudwatch_event_rule.container-stopped-rule: Updating CloudWatch Event Rule failed: InvalidEventPatternException: Event pattern is not valid. Reason: Match value must be String, number, true, false, or null at [Source: (String)"{"detail":{"containers":[{"exitCode":1}],"lastStatus":["STOPPED"],"stoppedReason":["Essential container in task exited"]},"detail-type":["ECS Task State Change"],"source":["aws.ecs"]}"; line: 1, column: 27] status code: 400

This is perplexing to me, since I'm following the exact structure laid out in the AWS CloudWatch documentation for containers. I've even attempted to put double quotes around 1 in case Terraform wants a string instead of a number.

I also tried to use AWS Console to manually edit the event pattern JSON, but received this error:

Validation error. Details: Event pattern contains invalid value (can only be a nonempty array or nonempty object)

I'm honestly a bit stumped at this point and would appreciate any tips on where my syntax is incorrect.

score 20 · Accepted Answer · answered Nov 01 '18 at 08:27

20

The event pattern syntax is pretty weird, I ran into the same issue. The following will work:

{
  "source": [
    "aws.ecs"
  ],
  "detail-type": [
    "ECS Task State Change"
  ],
  "detail": {
    "lastStatus": [
      "STOPPED"
    ],
    "stoppedReason": [
      "Essential container in task exited"
    ],
    "containers": {
      "exitCode": [
        1
      ]
    }
  }
}

I used $.detail.group in the Input Transformer to get the task family name in the notification message.

answered Nov 01 '18 at 08:27

Oscar Barrett

3,135
31
36

5

I'm not sure its possible, but woudn't `"exitCode": [ != 0]`, be a safer option? Or is the exit code guaranteed to be 0 or 1. – Joe.Zeppy Feb 22 '19 at 18:17
@Joe.Zeppy As far as I can tell at the moment there's no way to say doesn't match, although you could specify all possible values (`"exitCode": [ 1,2,3,4,...255 ]`) – Matthew Buckett Jan 24 '20 at 16:35
3

This is now possible `"exitCode": [ { "anything-but": 0 } ]` should work. – Manos Pasgiannis Jul 27 '20 at 10:58
2

@zoph here is a link to the docs if that helps https://docs.aws.amazon.com/eventbridge/latest/userguide/content-filtering-with-event-patterns.html#filtering-anything-but – Manos Pasgiannis Nov 08 '20 at 15:54
@zoph This errors out with the following message Event pattern contains invalid element (can only be Strings enclosed in quotes, numbers, and the unquoted keywords true, false, and null) any idea why? – dakrpssngr Aug 31 '21 at 15:07
Just to point out what is different in the example given in this answer: the value for "exitCode" is an array with the number 1 instead of just the number 1. All values in an event pattern must be an array (or an object if going to a more deeply nested level). See https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEventsandEventPatterns.html#ArraysInCloudWatchEventsPatterns – fenix.shadow Sep 13 '21 at 23:09
Thank you for this, doesn't really related to Terraform, this solution should appear in the docs, kudos – Meir Gabay Feb 21 '22 at 13:34

score 1 · Answer 2 · answered Oct 29 '18 at 09:33

As per https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEventsandEventPatterns.html, For a pattern to match an event, the event must contain all the field names listed in the pattern. The field names must appear in the event with the same nesting structure.

Can you try adding more fields lister here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html like clusterArn, containerInstanceArn etc?

Including an exit code 1 event in CloudWatch using Terraform for ECS

2 Answers2