AWS ECS starting a container is very slow after 1 day

Question

I'm using ECS scheduled task to run my cron jobs. I have around 6 jobs running at various times. 3 of my jobs run every 2 minutes. Everything works fine for the first day when I launch a new instance in the ECS cluster. Then after a day, I see lots of tasks are in a pending state. I ssh into my ECS EC2 instance and try to start a container manually using docker run command, it takes 4-5 minutes for the container to start. I have checked logs in /var/log/docker: I see this error

time="2018-07-23T01:06:46.688248652Z" level=error msg="Handler for GET /v1.21/containers/2f68db90950da30547c326780df5c90afd85495fa54d8b2fba7e2044a8c23024/json returned error: write unix /var/run/docker.sock->@: write: broken pipe" 2018-07-23 01:06:46.688296 I | http: multiple response.WriteHeader calls

I'm not sure if this is the cause of the issue. I can't think of any way to resolve this issue.

Please help.

Thank you

I'm curious: Are you running anything that uses /var/run/docker.sock directly (like DataDog)? — Brendan Long, Jul 30 '18 at 23:28
When i wrote this question, i wasn't using it, but I'm using it now. — shakyaabiral, Aug 01 '18 at 03:27
The issue still persists. I have set my autoscaling group to have 2 instances running as a temporary solution. — shakyaabiral, Aug 01 '18 at 03:33

Brendan Long · Answer 1 · 2018-08-01T13:03:18.730

0

I'm running into similar problems and opened a bug report against the ECS agent:

https://github.com/aws/amazon-ecs-agent/issues/1489

If you can add information about your setup there it could help. I'm still not sure what's wrong.

edited Aug 01 '18 at 13:03

answered Jul 30 '18 at 23:27

Brendan Long

53,280
21
146
188

AWS ECS starting a container is very slow after 1 day

1 Answers1