ECS awslogs Log group not being created for Task

Question

Here's part of my task definition:

"logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/main-frontend-production-php",
        "awslogs-region": "us-west-1",
        "awslogs-stream-prefix": "ecs",
        "awslogs-create-group": "true"
      }
    },

If I go view the task in the aws console it says the log driver is awslogs and to "View logs in cloudwatch" as soon as I click that I get an error: There was an error getting log events. The specified log group does not exist. and there are no logs.

Kind of at a loss, it seems like from this: https://docs.aws.amazon.com/AmazonECS/latest/userguide/using_awslogs.html that I'm doing it right?

Right now my tasks are stuck in "PENDING" and I have no logs to go off of to find out why.

How does the attached IAM roles policies look? – Chris Williams Jul 11 '20 at 14:56 — Chris Williams, Jul 11 '20 at 14:56

score 3 · Answer 1 · answered Jul 12 '20 at 05:50

The possible reason that causes this error as the error seems like the container instance able to get the list of Log group.

Make sure the container and log group are in the same region us-west-1.
Make sure the container has permission to create a log group
Just to narrow down the problem just create log-group /ecs/main-frontend-production-php using aws cli or aws console so you will know the actual reason as it because of region or permission.

aws logs create-log-group --log-group-name /ecs/main-frontend-production-php --region us-west-1

policy should look like

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        }
    ]
}

score 2 · Accepted Answer · answered Jul 14 '20 at 22:37

2

Ok I finally figured this one out. The reason I had no cloud watch logs was because the image was not getting pulled from ECR. I was always under the impression that "latest" was some magical docker tag. Apparently it's not and I have to actually tag it latest for it to find it.

I think the cloudwatch logs would have been working all along, but the image couldn't get pulled, so there were no logs to speak of.

answered Jul 14 '20 at 22:37

Brian Jenkins

349
1
6
22

1

Can you please explain more on this, How did you solve ! – Ezhilisai Apr 29 '21 at 20:02

score 1 · Answer 3 · answered Jul 11 '20 at 15:35

1

This will occur as the result of invalid permissions to write to CloudWatch logs.

You must attach permissions to the role you are using.

Follow the Using CloudWatch Logs with container instances documentation if you're unsure of how to do this.

answered Jul 11 '20 at 15:35

Chris Williams

32,215
4
30
68

Ok so I think I'm going in the right direction now, but still lost. That doc made it sound like I'm already supposed to have a role title `ecsInstanceRole` that was automatically created. I don't (using terraform maybe that's why). So I just created it, but they didn't specify what uses it? It almost sounds like it's just by convention the ECS agent will use a role with the name `ecsInstanceRole`. But I added the log policy there and the logs are still not generated. Should I add it to something? The task or the service or something? Sorry, new to AWS I'm a little lost. – Brian Jenkins Jul 13 '20 at 04:40

ECS awslogs Log group not being created for Task

3 Answers3