How to restart containers in AWS ECS?

Question

I have provided application configuration via consul's key-value store to the application containers running in ECS services.

The application reads its configuration from consul only once on start up.

When I need to change the configuration, how should I go about restarting the containers so that the application configuration is refreshed?

I am hoping to do this programmatically via the aws cli.

There's a number of answers here: https://serverfault.com/questions/705644/how-do-you-restart-all-tasks-of-a-service The popular solution seems to be to use the aws cli `aws ecs update-service --force-new-deployment ...` — WhyAyala, Oct 05 '21 at 14:20

score 40 · Accepted Answer · answered Apr 12 '17 at 23:20

40

You don't restart containers. You can however stop the individual tasks, and ECS will respawn another instance of your task somewhere on the cluster.

answered Apr 12 '17 at 23:20

Mitch Dempsey

38,725
6
68
74

I had considered. Would that not result in (minor) downtime, however? Be good to keep the ECS out-of-the-box blue-green deploys. – Nick Apr 12 '17 at 23:39
2

It would result in minor downtime, but so would a restart. You could increase the number of tasks for a service so that it scales up, remove the old task from the load balancer, and then stop the older task. – Mitch Dempsey Apr 13 '17 at 00:21
4

Downvoted. 1) It's not "you don't", it's "you shouldn't". 2) It's not an answer. You are just simply saying why it's a bad idea (which I totally agree), but you are not telling how to actually do this (a solution to perform that bad idea), if one wants to do it ever, for any reason. The same "you don't" instead of "you shouldn't", can apply to a good portion of StackOverflow questions when developers do things that are not common/normal/standard. And that kills being creative and a real problem-solver. – Aidin Jan 15 '21 at 20:18
1

@Aidin Are we reading the same answer? He does tell what to do instead. A lot of the issues are solved by telling the original poster that you don't do what the original poster wants. – Dragas Feb 21 '22 at 09:16
@dragas and which website do you recommend developers to ask their unorthodox technical questions? – Aidin Feb 22 '22 at 16:22
@Aidin would "you physically are unable to" be better for you? Because it's not a suggestion, it's actually impossible to restart a task. – Mitch Dempsey Feb 22 '22 at 22:09
It is physically possible. You can `ssh` there and then do `docker stop`. It's just the `start` that needs extra steps to work as expected. – Aidin Feb 24 '22 at 04:47
1

That would be using Docker, not "ECS". There is no restart feature in AWS ECS. Not sure why you continue being pedantic on this. – Mitch Dempsey Nov 06 '22 at 08:03

stemadsen · Answer 2 · 2022-01-05T13:49:05.560

Update:

As @Aidin mentioned, you can achieve it via the AWS CLI by forcing a new deployment like so:

aws ecs update-service \
--service <service name> \
--cluster <cluster name> \
--force-new-deployment \
[--profile guestapi-dev]

Note that this does not work on services with a CodeDeploy deployment controller.

Original answer:

I faced the same challenge, and what I did was follow this guide (using the old or new console depending on your service). I don't know if this can be done via the CLI, but it does actually "restart the service" in that it re-spawns new task(s) for your service and kills the old one(s).

In summary:

In the old console:

Find the service in the AWS console (ECS -> Cluster -> Service).
Click Update in the top right corner.
Check the ‘Force new deployment’ box.
Skip the other configurations and click Update Service.

In the new console:

Find the service in the AWS console (ECS -> Cluster -> Service).
Click Edit in the top right corner.
Expand Deployments options
Check the ‘Force new deployment’ box.
Click Update.

The service will re-deploy. You should be able to see the existing task(s) running, the new task(s) provision and lastly the old task(s) disappear.

It's "creating new containers" rather than "restarting the existing containers". Close enough though -- might work for some people, but not everyone. Thanks. — Aidin, Oct 13 '21 at 18:40

score 3 · Answer 3 · answered Jun 27 '18 at 11:58

3

this worked for me:

aws ecs list-tasks --cluster my-cluster-name | jq -r ".taskArns[]" | awk '{print "aws ecs stop-task --cluster my-cluster-name --task \""$0"\""}' | sh

answered Jun 27 '18 at 11:58

user326608

2,210
1
26
33

This restarts the whole task, not just one container. – matt2000 Oct 14 '19 at 19:52
Downvoted. 1) no explanation. 2) wrong. Wrong because what this snippet does is it stops the task, so if the task belongs to a service, the service is going to respawn "a new container" running the task. This has nothing to do, whatsoever, with restarting a container. – Aidin Jan 15 '21 at 19:54
1

If anyone is looking for the whole restart, this worked for me better than awk due to passing of parameters: ```aws ecs list-tasks --cluster "$ecs_cluster" | jq -r ".taskArns[]" | xargs -n1 aws ecs stop-task --no-cli-pager --cluster "$ecs_cluster" --task``` The --no-cli-pager prevents the output from stop-task from getting stuck after each one. – Baron Jan 17 '23 at 21:42

score 1 · Answer 4 · answered Oct 13 '21 at 14:24

1

Go to ECS dashboard. Just stop the running task from your ECS service from aws console. it'll spawn a new task and terminate the old one.

answered Oct 13 '21 at 14:24

Saravanan T

21
4

Similar to the other answer: it's "creating new containers" rather than "restarting the existing containers". Close enough though -- might work for some people, but not everyone. Thanks. – Aidin Oct 13 '21 at 18:47
Hi, the user explicitly asked `I am hoping to do this programmatically via the AWS CLI`. Also, can you [including an explanation](https://meta.stackoverflow.com/q/392712/5736950) of how and why this solves the problem. This would really help to improve the quality of your post and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please edit your answer to add explanations and give an indication of what limitations and assumptions apply. – Himanshu Bansal Oct 13 '21 at 19:00

score 1 · Answer 5 · answered Mar 29 '23 at 10:19

In the new console

Find the service in the AWS console (ECS -> Cluster -> Service)
Click on the {service name}
Click the "Update service" button in the top right corner.
Under the "Deployment configuration" section, Check the "Force new deployment" box.
Leave the other options as it is and Click Update.

score 0 · Answer 6 · answered Dec 27 '21 at 10:39

0

In conclusion, you cannot simply stop and start a container within the same task. You just start a new task. AWS should do a rolling bounce, so it will not give you a downtime, and new task will stay as long is is passing the health check

answered Dec 27 '21 at 10:39

kotsios geo

1

Garret Wilson · Answer 7 · 2023-08-30T16:19:46.763

Ideally using ECS Fargate you would have a different tag (i.e. unique version) for each image change, so that when you updated the CloudFormation template the cluster would redeploy automatically with the new image. If you're using the same version number e.g. for a SNAPSHOT version of the image and want to force the cluster to start using the new image, as the other answer to this question noted you can stop the cluster task(s), and when they restart they will automatically pull the latest image from the repository.

If you want to manually stop the task using the CLI, you'll need to first query the cluster for the task ARN, as explained by an answer to another question:

aws ecs list-tasks --cluster my-cluster --service my-service --output text --query taskArns[0]

(Note that this command as written only returns the ARN of the first task.)

Save this in a taskArn variable in Bash, and then use it with aws ecs stop-task to stop the task, specifying the task ARN using --task:

aws ecs stop-task --cluster my-cluster --task $taskArn --output text --query task.lastStatus

I added the --output text and --query task.lastStatus to the end to prevent pages of JSON cruft being shown. You may find other information to glean from it.

You can put it all together in a Bash script. Assuming you put $clusterName, $serviceName, etc. in variables, it might look like this:

taskArn=$(aws ecs list-tasks --profile $awsProfile --cluster $clusterName --service $serviceName --output text --query taskArns[0])
taskLastStatus=$(aws ecs stop-task --profile $awsProfile --cluster $clusterName --task $taskArn --output text --query task.lastStatus)
echo $clusterName $serviceName $taskArn $taskLastStatus

Note that the specifying --profile is optional if you're using the default configured profile.

score -3 · Answer 8 · answered Jan 15 '21 at 20:17

None of the two existing solutions on this question are satisfying. I don't have a full answer (yet), but I can A) tell you what I found, and B) tell you what is the "correct" architecture to handle this issue.

What I found

I was under the impression that SSHing into the instance and then simply docker restart <container-id> should work.

In fact, it initially seemed like it did. But, it turned out that I was wrong and it was just a can of worms waiting there for me! Doing so results in the container starting with no IAM role/credentials properly to talk to the other AWS services. My story in detail is on this Github issue of ecs-agent. It took me 10+ hours to find out that was the culprit. Apparently, containers will be in proper condition only if the ecs-agent starts them, and not you start/restart them.

What's the right way?

I believe the mentality and philosophy behind ECS/Tasks are that they want to take full control of the layer of abstraction between you and the running environment of the containers. You just say "Hey I want 3 of these user-avatar-uploader-to-s3 containers running" and it does that job for you. But you are not much welcome to meddle in the way they are doing their business!

However, if you want the containers to be configurable and pass certain params to it (e.g. the consul key-value pair in the original question), you are allowed to define them as Environment Variable both in the Task Definition (for each container) and in the Service/Task execution.

So, the right way would be to redo your container code to take these params (key-value pair) as Environment Variable (or from a configurable secure private S3 bucket, or AWS SecretsManager). Then put the desired values in the task/task-execution, and voila it should work. You can then change them at any time and ECS will take care of it. (Note that it will be a new container/task spinning up with the new settings, not your old one updated.)

That's it.

(I will update this answer as soon as I find how to do that emergency open-heart-surgery docker restarts.)

Downvoted. 1) Doesn't answer the question "How to restart containers in AWS ECS?" 2) why would you ever manually ssh to the instance to restart it? — WhyAyala, Oct 05 '21 at 14:15
1) The answer to "how to do you eat soup with fork" is "you don't. there are better alternatives. fork and soup don't work that way." 2) You need to get close to the bowl of soup anyway, if you want to eat it. :) Good luck! — Aidin, Oct 09 '21 at 18:31
1) Per your soup example, the answer is not "you don't", it's "you shouldn't". 2) Manually sshing in is inherently anti-pattern to iac. I'd even argue it's anti-pattern all around when working with cloud services. You could maybe do it in a test environment, but it's not a prod reusable solution. Hard stop. 3) MitchDempsey had an acceptable answer and follow up comment on handling downtime. Your answer not only walks through a bad practice, your actual solution just reiterates what OP has already said they've done. You've literally offered no solution here. You can keep your luck — WhyAyala, Oct 13 '21 at 17:12
I agree that I didn't exactly answer the original question per se; see the last line. My answer contains the best findings I had, including the Github Issue that's blocking it, and my comment there. I hoped it saves a few hours of anyone who wants to pick this up (until `ecs-agent` guys fix it.) Technically, this post could be a comment but since it was long and had links, I posted it as an answer for now. Once I finally see a solution to "restart the running containers" here, I'll take mine down. Until then, thanks for bearing with my "how-to-start" and "why" details here, Ayala! :) — Aidin, Oct 13 '21 at 18:46

How to restart containers in AWS ECS?

8 Answers8

Update:

Original answer:

What I found

What's the right way?