4

I am struggling to get a task running using ECS Fargate, and launched (ecs.runTask) from an AWS SDK script (JS/Node).

My current struggle is to get logs from the containers so that I can trouble shoot why they are stopping. I can't seem to get the Task Definition right so that they will be generated.

    logConfiguration: {
        logDriver: 'awslogs',
        options: {
            "awslogs-region": 'us-west-2',
            "awslogs-group": 'myTask',
            "awslogs-stream-prefix": "myTask",
            "awslogs-create-group": "true"
        }
    }

I have set the log driver for them to awslogs, but when I try to view the logs in CloudWatch, I get various kinds of nothing:

  • If I specify the awslogs-create-group as "true" (it requires a string, rather than a Boolean, which is strange; I assume case doesn't matter), I nevertheless find that the group is not created.
  • If I create the group manually, I find that the log stream is not created.

I suspect that there may be an error in my permissions, though of course there is no error messaging to confirm. The docs (here) indicate that I need to attach certain policies to ecsInstanceRole, which seems to be a placeholder for a role that is used somewhere in the process.

But I have attached such a policy to my ECS executionRole, to the role that executes my API call to runTask, and I have looked for any other role that might be involved (an actual "instanceRole" doesn't seem to exist in the Task Def), and nothing is improving my situation.

I'd be happy to supply more information, but at this point I'm not sure where my blind spot is.

Can anyone see it?

TwainJ
  • 1,187
  • 1
  • 13
  • 26
  • Verify your task execution policy has "logs:CreateLogGroup" permission. – Haran Nov 15 '19 at 05:38
  • Thank you @Haran, I did check and it is there. I created a custom policy with that permission, and then added that, and the AWS managed, `AmazonECSTaskExecutionRolePolicy `, which has the other necessary permissions (as far as I can tell), such as `logs:CreateLogStream`. – TwainJ Nov 15 '19 at 21:19
  • If all required permissions are in place then try to troubleshoot other areas. 1) See whether your image doesn't have any issues. 2) Go to the 'Stopped' task and then check the reason why it got failed (in the AWS Console it would be available). There could be other reasons also for a task to fail. Ignore if you have already done these steps. – Haran Nov 16 '19 at 02:59
  • Thanks @Haran, the errors there were less than helpful - that's why I was hoping to get logging working. I had to jump to another priority, but I will get back to this as soon as I can to follow up. – TwainJ Nov 19 '19 at 23:26
  • @TwainJ I am facing the exact same issue, would be really grateful if you can tell how you managed to solve the issue – Gaurav Jul 16 '20 at 14:07

1 Answers1

1

Go to your Task Definition. You should find a section called "Task execution IAM role". The description says -

This role is required by tasks to pull container images and publish container logs to Amazon CloudWatch.

The role you attach here needs a policy like AmazonECSTaskExecutionRolePolicy (AWS managed policy), and the Trusted Entity is ecs-tasks.amazonaws.com.

Also, the awslogs option awslogs-create-group is not needed, I think.

Soubhik Mondal
  • 2,666
  • 1
  • 13
  • 19
  • I do have an execution role defined (the task definition member is `executionRoleArn`). That's the role I mentioned that I have assigned policies per that link (it was the `AmazonECSTaskExecutionRolePolicy`, in fact). If we can say for certain that that is the role the link refers to as `ecsInstanceRole`, then maybe we can eliminate permissions as the problem. I'm not sure where else to look, though. – TwainJ Nov 13 '19 at 04:12
  • Docs for [Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html). – Soubhik Mondal Nov 13 '19 at 04:32
  • I think a separate ECS Instance Role would come into play if this were not Fargate. [Docs here](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html). – Soubhik Mondal Nov 13 '19 at 04:34