How can I update and ECS service adding an addition load balance to which a service talks to without downtime?

Question

We use terraform to manage our AWS resources and have the code to change the service from one to two load balancers.

However, terraform wants to destroy the service prior to recreating it. AWS cli docs indicate the reason - the API can only modify LBs during service creation and not on update.

• create-service
• update-service

It seems we need a blue/green deploy with the one LB and two LB services existing at the same time on the same cluster. I expect we'll need to create multiple task sets and the rest of the blue/green approach prior to this change (we'd planned for this anyway just not at this moment)

Does anyone have a good example for this scenario or know of any other approaches beyond full blue/green deployment?

Answer 1

Alas, it is not possible to change the number of LBs during an update. The service must be destroyed and recreated.

Ideally, one would be doing blue green deploys with multiple ECS clusters and a set of LBs. Then cluster A could have the old service and cluster B have the new service allowing traffic to move from A to B as we go from blue to green.

We aren't quite there yet but plan to be soon. So, for now, we will do the classic parking lot switch approach:

In this example the service that needs to go from 1 LB to 2 LBs is called target_service

1. clone target_service to become target_service2
2. deploy microservices that can talk to either target_service or target_service2
3. verify that target_service and target_service2 are both handling incoming data and requests
4. modify target_service infra-as-code to go from 1 to 2 LBs
5. deploy modified target_service (terraform deployment tool will destroy target_service leaving target_service2 to cover the gap, and then it will deploy target_service with 2 LBs
6. verify that target_service with 2 LBS works and handle requests
7. destroy and remove target_service2 as it is no longer needed

So, this is a blue-green like deploy albeit less elegant.

Answer 2

To update an Amazon ECS service with multiple load balancers, you need to ensure that you are using version 2 of the AWS CLI. Then, you can run the following command:

aws ecs update-service \
    --cluster <cluster-name> \
    --service <service-name> \
    --load-balancers "[{\"containerName\": \"<container-name>\", \"containerPort\": <container-port>, \"targetGroupArn\": \"<target-group-arn1>\"}, {\"containerName\": \"<container-name>\", \"containerPort\": <container-port>, \"targetGroupArn\": \"<target-group-arn2>\"}]"

In the above command, replace with the name of your ECS cluster, with the name of your ECS service, with the name of the container in your task definition, with the port number the container listens to, and and with the ARN of your target groups.

Answer 3

Update 2022

ECS now supports updation of load balancers of a service. For now, this can be achieved using AWS CLI, AWS SDK, or AWS Cloudformation (not from AWS Console). From the documentation:

When you add, update, or remove a load balancer configuration, Amazon ECS starts new tasks with the updated Elastic Load Balancing configuration, and then stops the old tasks when the new tasks are running.

So, you don't need to create a new service. AWS update regarding this here. Please refer Update Service API doc on how to make the request.