9

Code that uses the AWS Node SDK doesn't seem to be able to gain the role permissions of the ECS task.

If I run the code on an EC2 ECS instance, the code seems to inherit the role on the instance, not of the task.

If I run the code on Fargate, the code doesn't get any permission.

By contrast, any bash scripts that run within the instance seem to have the proper permissions.

Indeed, the documentation doesn't mention this as an option for the node sdk, just:

  1. Loaded from IAM roles for Amazon EC2 (if running on EC2),
  2. Loaded from the shared credentials file (~/.aws/credentials),
  3. Loaded from environment variables,
  4. Loaded from a JSON file on disk,
  5. Hardcoded in your application

Is there any way to have your node code gain the permissions of the ECS task?

This seems to be the logical way to pass permissions to your code. It works beautifully with code running on an instance.

The only workaround I can think of is to create one IAM user per ECS service and pass the API Key/Secret as environmental variables in the task definition. However, that doesn't seem very secure since it would be visible in plain text to anyone with access to the task definition.

Cristian Vrabie
  • 3,972
  • 5
  • 30
  • 49
  • How are you creating the client, what function are you using? – titogeo Sep 29 '18 at 15:46
  • 1
    @titogeo just instantiating the clients with no parameters which should use the credentials chain resolver – Cristian Vrabie Sep 30 '18 at 16:15
  • https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_IAM_role.html hope you are following this – titogeo Sep 30 '18 at 18:06
  • 1
    Can you edit the question to include the output of `aws ecs describe-task-definition` of the task and also the output of `aws iam get-role` for the role your task is assuming? The AWS SDK will use ECS task roles transparently as long as these and your ECS agent are properly configured (or just automatically when using Fargate) so I guess it's something to do with your task or IAM role configuration that is wrong/missing. – ydaetskcoR Oct 01 '18 at 13:29
  • @ydaetskcoR thanks for looking into this. [task definition](https://gist.github.com/cvrabie/35465435fcdd2f64a6c1763b4db573f6) and [role](https://gist.github.com/cvrabie/bcb6b62bbf3250c914053f0c15352be9) as requested. – Cristian Vrabie Oct 08 '18 at 08:15

2 Answers2

2

Your question is missing a lot of details on how you setup your ECS Cluster plus I am not sure if the question is for ECS or for Fargate specifically.

Make sure that you are using the latest version of the SDK. Javascript supports ECS and Fargate task credentials.

Often there is confusion about credentials on ECS. There is the IAM role that is assigned to the Cluster EC2 instances and the IAM role that is assigned to ECS tasks.

The most common problem is the "Trust Relationship" has not been setup on the ECS Task Role. Select your IAM role and then the "Trust Relationships" tab and make sure that it looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

In addition to the standard Amazon ECS permissions required to run tasks and services, IAM users also require iam:PassRole permissions to use IAM roles for tasks.

Next verify that you are using the IAM role in the task definition. Specify the correct IAM role ARN in the Task Role field. Note that this different than Task Execution Role (which allows containers to pull images and publish logs).

Next make sure that your ECS Instances are using the latest version of the ECS Agent. The agent version is listed on the "ECS Instances" tab under the right hand side column "Agent version". The current version is 1.20.3.

Are you using an ECS optimized AMI? If not, add --net=host to your docker run command that starts the agent. Review this link for more information.

John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • I can confirm that I have both the proper trust relationship and the `iam:PassRole` permission. The question is for `Fargate` specifically. Same role works fine on EC2 instances. The network mode is `awsvpc` and as far as I can tell, I can't control the parameters of `docker run` in Fargate. – Cristian Vrabie Oct 05 '18 at 10:58
  • Upon closer inspection I can see that the `AWS JS SDK` calls are failing with `Error: connect EINVAL 169.254.169.254:80 - Local (0.0.0.0:0)`. I think that's the AWS metadata service so I'm assuming that the `AWS JS SDK` is trying to fetch the credentials and failing. As previously said, a bash script running in the same container with the exact same settings can use the task role credentials perfectly. – Cristian Vrabie Oct 05 '18 at 11:00
  • PS: Sorry for the slow reply. The bounty has expired but this is such an important problem for us that I'm happy to open a new, bigger one. – Cristian Vrabie Oct 05 '18 at 11:01
  • I will look into your issue again after work today. This is a metadata access problem and should be easy to figure out now that I know the issue is with Fargate. – John Hanley Oct 05 '18 at 15:55
  • This is incredibly weird. My start command first runs a bash script that fetches a config file from S3 then runs node. `"start": "/var/get-config.sh && node app.js"`. I'm wondering if something is happening there so that node doesn't get certain environmental variables. Any code in `get-config.sh` has all the task permission and can call any aws service. I've added a curl to http://169.254.170.2/v2/metadata in there and it works just fine. – Cristian Vrabie Oct 11 '18 at 10:11
1

I figured it out. This was a weird one.

A colleague thought it would be "safer" if we call Object.freeze on proccess.env. This was somehow interfering with the SDK's ability to access the credentials.

Removed that "improvement" and all is fine again. I think the lesson is "do not mess with process.env".

Cristian Vrabie
  • 3,972
  • 5
  • 30
  • 49