CannotPullContainerError in ECR when trying to use VPC Endpoints

Question

I have an ECS stack made with AWS CDK.

import * as cdk from 'aws-cdk-lib'
import { Construct } from 'constructs'
import * as ec2 from 'aws-cdk-lib/aws-ec2'
import * as ecs from 'aws-cdk-lib/aws-ecs'
import path = require('path')

export class CdkPlaygroundEcsAuroraStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)


    // VPC
    const vpc = new ec2.Vpc(this, 'ecs-vpc', {
      maxAzs: 2,
      subnetConfiguration: [
        {
          name: 'public',
          subnetType: cdk.aws_ec2.SubnetType.PUBLIC
        },
        {
          name: 'private',
          subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED
        }
      ]
    })


    // VPC Endpoints
    const ecrVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ECRVpcEndpoint', {
      vpc,
      service: ec2.InterfaceVpcEndpointAwsService.ECR,
      privateDnsEnabled: true
    })

    const cloudWatchVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'cloudWatchVpcEndpoint', {
      vpc,
      service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH,
      privateDnsEnabled: true
    })

    const s3GatewayEndpoint = new ec2.GatewayVpcEndpoint(this, 'S3GatewayEndpoint', {
      service: ec2.GatewayVpcEndpointAwsService.S3,
      vpc,
      subnets: [{ subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED }]
    })


    // ECS Cluster
    const ecsCluster = new ecs.Cluster(this, 'ecs-cluster', {
      vpc,
      enableFargateCapacityProviders: true,
      clusterName: 'ecs-cluster'
    })

    const taskDefinition = new ecs.FargateTaskDefinition(this, 'task1', {
      cpu: 256,
      memoryLimitMiB: 512
    })

    const container = taskDefinition.addContainer('container', {
      image: ecs.ContainerImage.fromAsset(path.resolve(__dirname, '../server')),
      memoryLimitMiB: 256
    })

    const service = new ecs.FargateService(this, 'server', {
      cluster: ecsCluster,
      taskDefinition,
      serviceName: 'server',
      vpcSubnets: {
        subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED
      }
    })
  }
}

It contains an ECS Task running in a private subnet without a NAT Gateway. I want to pull the image via the configured VPC Interface Endpoint. According to the documentation, I have setup VPC Endpoints for ECR, S3 and Cloudwatch.

But the task can not pull the image for the container with the following error:

CannotPullContainerError: pull image manifest has been retried 5 time(s): failed to resolve ref 650289367947.dkr.ecr.eu-central-1.amazonaws.com/cdk-hnb659fds-container-assets-650289367947-eu-central-1:d71b37a3ce0b63a08136ce5b816ea2d5d4b677fc8e3ee8b6eda9738d7c16ebb1: failed to do request: Head "https://650289367947.dkr.ecr.eu-central-1.amazonaws.com/v2/cdk-hnb659fds-container-assets-650289367947-eu-central-1/manifests/d71b37a3ce0b63a08136ce5b816ea2d5d4b677fc8e3ee8b6eda9738d7c16ebb1": dial tcp 3.121.190.14:443: i/o timeout

To me it looks like ECS is trying to access ECR over the internet (dial tcp 3.121.190.14:443). Why doesn't ECS use the VPC Interface Endpoint and how can I change this to fix the error?

Tobias S. · Accepted Answer · 2023-07-23T00:44:40.627

To access ECR from ECS via VPC Endpoints, you require 3 endpoints. If you are using Fargate Version > 1.4.0, you will need the following endpoints:

com.amazonaws.region.ecr.dkr to access the Docker Registry APIs
com.amazonaws.region.ecr.api to access the Amazon ECR API
com.amazonaws.region.s3 to download the images from S3

If you also wish to enable CloudWatch logging, you will additionally need the com.amazonaws.region.logs endpoint.

The complete cdk definition would look like this:

// access ECR
new ec2.InterfaceVpcEndpoint(this, 'ECRVpcEndpoint', {
  vpc,
  service: ec2.InterfaceVpcEndpointAwsService.ECR,
  privateDnsEnabled: true
})
new ec2.InterfaceVpcEndpoint(this, 'ECRDockerVpcEndpoint', {
  vpc,
  service: ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER,
  privateDnsEnabled: true
})
new ec2.GatewayVpcEndpoint(this, 'S3GatewayEndpoint', {
  service: ec2.GatewayVpcEndpointAwsService.S3,
  vpc,
  subnets: [{ subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED }]
})

// access Cloudwatch logging
new ec2.InterfaceVpcEndpoint(this, 'CloudWatchLogsVpcEndpoint', {
  vpc,
  service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS,
  privateDnsEnabled: true
})

Docs

CannotPullContainerError in ECR when trying to use VPC Endpoints

1 Answers1