20

The task definition my Service uses is pulling the "latest" tagged version of my image.

When I update my service though and "force new deployment" i look at the events and see this:

service MYSERVICE was unable to place a task because no container instance met all of its requirements. The closest matching container-instance .... is already using a port required by your task

I then went to my cluster and stopped all tasks.

Then went back to my Service and updated with force new deploy again. This seems to have worked

Will I have to stop all tasks and update the service each time I want to deploy a new image? Or is there a "right" way to do this?

EDIT: So if I stop one task the service will automatically replace it. So I can update my service and "force new deploy" then stop one task at a time to get a sort of a rolling update. Not sure if there is a feature to automate that beyond my own scripting

Just to follow up, as stated in the answers I just needed to use dynamic port mapping.

Initially, when I first started I didn't have a load balancer so I was hitting the EC2 instances directly to access the running containers. Of course in order to do this I had to expose a static port on the EC2 host.

I added a load balancer but kept that static port mapping not understanding how dynamic port mapping worked. All I had to do was change my task definition to set the host port to "0". Now I have no static port mappings on the hosts, the NLB does the routing for me and deploys work as expected

red888
  • 27,709
  • 55
  • 204
  • 392

5 Answers5

36

While the other answers are correct, I don't think they apply to the problem you have. I say this because it's a problem my team has faced as well, and doesn't really have anything to do with trying to launch multiple containers on the same instance - if I understand correctly, you're just trying to replace the existing container from an updated task definition. If you want to put multiple copies of the same container on a single box, definitely look at the suggestions from the other answers (in addition to the details below), but for rolling deploys, dynamic ports are by no means required.

[[ Side note for completeness: it's possible that your forced deploy threw the error you posted because it just takes a while for EC2 to clean up resources stopped by ECS. You'll see the same sort of issue if you're trying to force stop / start a task -- we've seen similar errors when trying to restart a container that was configured to allocate >50% of the available instance memory. You'll get those types of resource errors until the EC2 instance is completely cleaned up, and reported back to ECS. I've seen this take upwards of 5 minutes. ]]

To your question then, unfortunately for now there aren't any great built-in mechanics from AWS for performing a rolling restart of tasks. However, you can do rolling deploys.

As you're probably aware already, your Service relies on a task definition that's specified. Note that it's reliant on the task definition number, and doesn't care about the container tags in the way that the EC2 instance will.

The below settings are where the magic happens for enabling rolling deploys; you can find these configuration options in your service settings.

magic

For you to be able to do rolling deploys, you have to have at least 2 tasks running.

  • Number of tasks -- The number of tasks your service wants to run (n)
  • Minimum healthy percent -- The minimum healthy % of n when deploying new tasks
  • Maximum percent -- The maximum % of n that can be added when deploying new tasks

So for a real example, let's assume you have the following configuration:

Number of tasks: 3
Minimum healthy percent:  50
Maximum percent: 100

If you change the task definition that your service is pointing at, it will initiate a rolling deploy. We have 3 running tasks, but allow for >=50% healthy. ECS will kill one of your tasks, making the healthy % drop to 66%, still above 50%. Once the new task comes up, the service is again at 100%, and ECS can continue with rolling the deploy to the next instance.

Likewise, if you had a configuration where minimum % == 100, and maximum % == 150 (assuming you have capacity), ECS will launch an additional task; once it's up, you have a healthy percent of 133%, and it can safely kill one of the old tasks. This process continues until your new task is fully deployed.

MrDuk
  • 16,578
  • 18
  • 74
  • 133
  • 3
    this answer so far the most suitable in my case, it's nice to understand how the deployment worked. I don't want to use a load balancer and only have 1 container running. So I set up `minimum healthy percent == 0` – hakuna1811 Jul 19 '19 at 14:34
  • This made it work! This is my first time with ECS and of course I set it to 1 task. MH% was set to 100 and thereof the task was never being terminated to be updated. Not warning the user about this is just poor UX! – Mauro Oct 16 '19 at 14:10
  • This helped me to bring a lot of clarification into the topic. Right now I'm wondering how it works with one task. I did set the number of tasks and the minimum healthy percent to 0 but the cluster doesn't make an effort to update the machines. Do you have any idea how to trigger that? – patreu22 Mar 29 '20 at 10:28
  • Is your maximum set to 200? That would allow you to bring up an extra task and then drain off the old one. – MrDuk Apr 03 '20 at 17:34
  • We have ours set to 40% minimum and 200% maximum and we are still getting this issue. We have the `portMappings.hostPort` in the task definition hard coded, because we have to have it static in order to allow prometheus to discover and pull metrics. Any chance I am missing something else? – Jeremy Jun 18 '20 at 21:10
7

When using ECS (or any other orchestrator) it is encouraged that you use Dynamic Port Mapping.

Basically ECS will assign a random unassigned port to your container. ECS then offers ways to retrieve that port number, using the agent instrospection API or the docker client itself. However I wouldn't try to retrieve the port, I would instead rely on an Application Load Balancer (ALB) which allows you to use a single endpoint to access any targeted containers independently of its dynamically assigned port. When updating your service the ALB will seamlessly transition to the newest version of the container without any disruption.

Finally, inside the container the local port will remain the same so you don't have to handle things differently.

Laurent Jalbert Simard
  • 5,949
  • 1
  • 28
  • 36
  • 3
    Dynamic port mapping is **neither encouraged nor best practice**. It just depends on your use case. – Matteo Jul 11 '20 at 08:27
5

Without dynamic ports, only one instance of a service can be deployed per container as the port being used by the instance cannot be used by any other instance. When you update the service, it will try to restart all its instances and if more than one instances are being started on a single EC2 container, the startup will fail.

Better to use docker containers with dynamic port mapping in ECS cluster.

S.K.
  • 3,597
  • 2
  • 16
  • 31
4

I had also faced the same issue when new instance of same microservice(container) starts up due to autoscaling, As ports for each microservice are fixed and configured in application.yml and when same service gets up on same EC2 due to autoscaling then it tries to acquire the same port that is already used by its previous instance(e.g if I am having X as microservice running on port 3102, if another instance of same service gets up due to aotoscaling then it may gets up on different EC2 machine or on same EC2 machine in ECS Cluster but ECS decides where to launch new instance of microservice based on availability of CPU and RAM, if new instance gets up on different EC2 machine then no issues because port will be free, but if instance will get up on same ec2 machine that it will not start and shout that port is already in use)

For solving this we need to update the Task definitions of our ECS Cluster only, no change is need in Images, Have to enable Dynamic port mapping in task definitions so that any number of microservice instance can run on same Ec2 machine

Without Dynamic Port Mapping Task Definition configurations will be

  1. Network Mode - Host
  2. In Port Mapping, Container Port will be : Port of your application configured in application.yml

enter image description here

enter image description here

For Enabling Dynamic Port Mapping in ECS

  • Change Network Mode : Bridge
  • Configure Port Mapping, Make the Host Port to 0 and container port equivalent to port of your application configured in application.yml

enter image description here enter image description here

ABHAY JOHRI
  • 1,997
  • 15
  • 19
1

Without a doubt, this is the best option because the ALB is slow and flawed. thank you @erncnerky

Firstly, set desired count as 0:

aws ecs update-service --cluster cluster_name --service service_name --desired-count 0

After the below command, you can dynamically check the number of running container instance:

aws ecs describe-services --cluster cluster_name --service service_name

Then run the below command:

aws ecs update-service --cluster cluster_name --service service_name --desired-count 1
Danny G
  • 3,660
  • 4
  • 38
  • 50
Eric Ricardo
  • 11
  • 1