How to configure AWS ECS Fargate Tasks/Cluster to access private Docker Container Registry (like GitHub Container Registry) with Pulumi?

Question

I'am working on a AWS Fargate Cluster setup with Pulumi and my current program already successfully creates a Cluster incl. Fargate Tasks that run a public accessible container image. My image is based on Spring Boot (project code on GitHub):

import * as awsx from "@pulumi/awsx";

// Spring Boot Apps port
const port = 8098;

// Create a ApplicationLoadBalancer to listen for requests and route them to the container.
const alb = new awsx.lb.ApplicationLoadBalancer("fargateAlb");

const albTargetGroup = alb.createTargetGroup("fargateAlbTargetGroup", {
    port: port,
    protocol: "HTTP",
    healthCheck: {
        // Use the default spring-boot-actuator health endpoint
        path: "/actuator/health"
    }
});

const albListener = albTargetGroup.createListener("fargateAlbListener", { port: port, protocol: "HTTP" });

// Define Container image published to the GitHub Container Registry
const service = new awsx.ecs.FargateService("microservice-api-spring-boot", {
    taskDefinitionArgs: {
        containers: {
            microservice_api_spring_boot: {
                image: "ghcr.io/jonashackt/microservice-api-spring-boot:latest",
                memory: 768,
                portMappings: [ albListener ]
            },
        },
    },
    desiredCount: 2,
});

// Export the URL so we can easily access it.
export const apiUrl = albListener.endpoint.hostname;

Everything works as expected, but now I need to use a private container image ghcr.io/jonashackt/microservice-api-spring-boot-private:latest instead. Switching my code to use the new image, my Fargate Cluster Service keeps stopping and starting new Tasks/Containers endlessly. Looking into my Fargate Cluster's Service' Task tab and switching to the Stopped Task status I see lot's of STOPPED (CannotPull... errors like this:

If I click into one of the stopped Tasks I see the following Stopped reason:

CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref "ghcr.io/jonashackt/microservice-api-spring-boot-private:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized

So how can I configure access to a private Container Registry (here GitHub Container Registry) using Pulumi?

score 6 · Accepted Answer · answered May 10 '21 at 11:30

In the AWS docs there's a detailed guide on how to grant ECS EC2 & Fargate launch type Tasks access to private Registries. Derived from that there are 4 steps to take:

Obtain Token or credentials to access private Container Registry
Create AWS Secrets Manager Secret containing Token/Creds to private Registry
Craft Task Execution Role (using aws.iam.Role) containing inlinePolicy for private Container Registry access
Enhance awsx.ecs.FargateService to use our Task Execution Role & Secret ARN in repositoryCredentials

0. Obtain Token or credentials to access private Container Registry

If you don't already have them, you'll need to create an Access Token or credentials inside our private Registry so that an external service is able to access it. With GitHub Container Registry for example we need to create a Personal Access Token (see docs here) using the read:packages scope as a minimum:

1. Create AWS Secrets Manager Secret containing Token/Creds to private Registry

Now head over to the AWS Secrets Manager console at https://console.aws.amazon.com/secretsmanager/ and create a new Secret via the Store a new secret button. In the GUI choose Other type of secrets and Plaintext - and then fill in your private Registry credentials as JSON:

{
  "username": "yourGitHubUserNameHere",
  "password": "yourGitHubPATHere"
}

Select Next and provide a Secret Name like githubContainerRegistryAccess. Hit Next again and leave Disable automatic rotation as the default. Next again and hit Store to create the Secret. Finally copy the Secret ARN like arn:aws:secretsmanager:awsRegionHere:yourAccountIdHere:secret:githubContainerRegistryAccess-randomNumberHere to your notepad or editor for later reference.

2. Craft Task Execution Role (using aws.iam.Role) containing inlinePolicy for private Container Registry access

As the docs tell us we need to add the permission to access the Secrets Manager Secret as Inline Policy to the Fargate Tasks Execution Role. Now as the new awsx.ecs.FargateService automatically creates such a Task Execution Role, but we can't access it afterwards really, we need to create the whole thing ourselves. To create the aws.iam.Role we can have a look into the Pulumi docs about it. A Pulumi aws.iam.Role consists of multiple components, and we need 3 of them:

assumeRolePolicy: I didn't really wanted to define this myself, but the aws.iam.Role can't be created without it. It's crucial to choose "sts:AssumeRole" as the Action and "ecs-tasks.amazonaws.com" as the Service Principal.
inlinePolicies: This array will take our InlinePolicy which the Task needs to access the Secrets Manager Secret (aka the whole point of this Role creation here). It's exactly the same as described in the AWS docs - really keep an eye on the correct arn names inside Resources. One of the two is exactly our Secrets Manager Secret ARN!
managedPolicyArns contains the default Policies attached to a Fargate Task by Pulumi (I simply had a look into the AWS Console to find their arns).

Here's the Pulumi code we need for the InlinePolicy to be defined correctly:

const taskExecutionRole = new aws.iam.Role("microservice-api-spring-boot-execution", {
    assumeRolePolicy: {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Principal": {
                    "Service": "ecs-tasks.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }, inlinePolicies: [
        {
            name: "ghcr-secret-access",
            policy: JSON.stringify({
                Version: "2012-10-17",
                Statement: [
                    {
                        Effect: "Allow",
                        Action: [
                            "kms:Decrypt",
                            "secretsmanager:GetSecretValue"
                        ],
                        Resource: [
                            "arn:aws:secretsmanager:awsRegionHere:yourAccountIdHere:secret:githubContainerRegistryAccess-randomNumberHere",
                            "arn:aws:kms:awsRegionHere:yourAccountIdHere:key/key_id"
                        ]
                    }]
            })
        },
    ],
    managedPolicyArns: [
        "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
        "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
    ]
});

3. Enhance awsx.ecs.FargateService to use our Task Execution Role & Secret ARN in repositoryCredentials

Now this is the final step. We need to use our created Task Execution Role and attach it to our awsx.ecs.FargateService using the executionRole parameter. And we also need to provide our Secrets Manager Secret ARN (again) as repositoryCredentials:credentialsParameter to our awsx.ecs.FargateService. This looks somehow like this:

// Define Container image published to the GitHub Container Registry
const service = new awsx.ecs.FargateService("microservice-api-spring-boot", { 
    taskDefinitionArgs: {
        containers: {
            blueprint_helloworld: {
                image: "ghcr.io/jonashackt/microservice-api-spring-boot-private:latest",
                memory: 768,
                portMappings: [ albListener ],
                // Access private GitHub Container Registry: we need to provide the Secret ARN as repositoryCredentials
                // see https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/awsx/ecs/#Container-repositoryCredentials
                repositoryCredentials: {
                    credentialsParameter: "arn:aws:secretsmanager:awsRegionHere:yourAccountIdHere:secret:githubContainerRegistryAccess-randomNumberHere",
                }
            },
        },
        executionRole: taskExecutionRole,
    },
    desiredCount: 2,
});

Now a pulumi up should bring up your Fargate Tasks as expected, since they're now able to pull the container images from the private GitHub Container Registry. Inside the AWS ECS Cluster view you should see your running Tasks:

how do you do this when you use the GITHUB_TOKEN rather than generating your personal token in GitHub? — rpmansion, Oct 30 '21 at 17:26
Accessing the GitHub Container Registry from *within* GitHub Actions has been made easier where you don't need to generate a PAT anymore and can directly use the `GITHUB_TOKEN`. But AFAIK to access the GitHub Container Registry from *outside* of GitHub Actions context (e.g. here from AWS Fargate), you still need the PAT. — jonashackt, Nov 08 '21 at 12:27

How to configure AWS ECS Fargate Tasks/Cluster to access private Docker Container Registry (like GitHub Container Registry) with Pulumi?

1 Answers1