1

I have an existing service with 2 tasks running. I updated the existing service with a new task definition and the old one was dergistered. The new task failed to come up but the old tasks were killed and the service tried to bring up the new tasks but it failed. This leads to a situation when there are no tasks running at all. The reason the new tasks were not able to come up was because of a bug in my code. But, shouldn't ECS have ensured that the old healthy tasks were still running when the new ones where not up.

When does ECS decides to kill the old tasks? I understand it is when the task status goes to "Running". Does "Running" means, it is healthy? On further debugging, I found that the problematic task was in running state for some time.

Sunil Kumar Mohanty
  • 371
  • 4
  • 14

2 Answers2

2

Well, let me explain what is going on:

When you update your Task Definition (of which you assumingly have some tasks running), ECS tries to execute a blue-green deployment, i.e. wait to have new tasks in ready state and then stop the old ones.

Now, your task will enter ready state when the container boots, but when it tries to execute some code which has an error inside, it will crash and die, so it will change its state again, but it has already passed through ready state and so the old tasks have already been killed.

AlexK
  • 1,380
  • 10
  • 17
  • Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail) – Sunil Kumar Mohanty Nov 20 '18 at 12:49
  • Refer to this https://stackoverflow.com/questions/51425983/ensure-ecs-only-kills-old-tasks-when-new-ones-are-ready, I think it might help. – AlexK Nov 20 '18 at 12:53
  • Have you had any success? – AlexK Nov 20 '18 at 14:47
  • I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB. – Sunil Kumar Mohanty Dec 07 '18 at 08:22
0

I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are marked as healthy by ALB.

Sunil Kumar Mohanty
  • 371
  • 4
  • 14