How can I remove shutdown Docker Service tasks after a rolling update?

Question

I'm using Docker Swarm to test services on AWS. I recently applied an update to the service like this:

 docker service update --image TestImage:v2 --update-parallelism 2 \
--update-delay 10s TestService2

The update worked as intended, and the service updated the task containers to v2. However a quick docker service ps TestService2 | grep "v1" reveals a bunch of shutdown TestImage:v1 tasks.

a0w77kj0k6jfg4r9g4nz47zzg   \_ TestService2.1   TestImage:v1  W1  Shutdown       Shutdown 36 minutes ago
2of4mc63ekzbib01w3x7q6sdm   \_ TestService2.2   TestImage:v1  W2  Shutdown       Shutdown 37 minutes ago
495frrpza5pxt205o1594x54a   \_ TestService2.3   TestImage:v1  W1  Shutdown       Shutdown 36 minutes ago
57l0gsqd26u2e5gdj30w8mcn9   \_ TestService2.4   TestImage:v1  M1  Shutdown       Shutdown 36 minutes ago
baoe1i79fswb34ydwbpafg6tm   \_ TestService2.5   TestImage:v1  M3  Shutdown       Shutdown 35 minutes ago
3uxi7kwxb73z69km6s17son58   \_ TestService2.6   TestImage:v1  M2  Shutdown       Shutdown 37 minutes ago
99cg4arnt1y52nd8d422bdu49   \_ TestService2.7   TestImage:v1  M3  Shutdown       Shutdown 36 minutes ago
cq5716jqp40h6jugo1j9ilzwp   \_ TestService2.8   TestImage:v1  M1  Shutdown       Shutdown 35 minutes ago
awlz1kxbrjk51dey7frm14d8u   \_ TestService2.9   TestImage:v1  W3  Shutdown       Shutdown 35 minutes ago
4xdi9a1jweyehfqlt76uynf3i   \_ TestService2.10  TestImage:v1  M2  Shutdown       Shutdown 36 minutes ago
eo4t6i0gaj5i294fcdnb3qncq   \_ TestService2.11  TestImage:v1  W3  Shutdown       Shutdown 35 minutes ago
3ydeuxdjquulv5xj94b9ovuwu   \_ TestService2.12  TestImage:v1  W1  Shutdown       Shutdown 36 minutes ago

How can I remove these without going to each individual swarm node and running docker rm on the exited containers? I don't think theres a docker service command to do it, I've looked through the docs, but does anyone know of a hack or script that I can run on a Swarm Manager to clean up the nodes?

Thanks!

Geige V · Accepted Answer · 2017-09-20T20:35:30.730

28

The containers for those services are removed after a rolling update; you are simply left with a log of those that were shutdown.

You can limit the number you see using

docker swarm update --task-history-limit 5

edited Sep 20 '17 at 20:35

answered Jan 20 '17 at 01:15

Geige V

1,389
1
16
28

5

How long after the update are they removed? If I log into a node, and I do a `docker ps -a` I see the container still there but exited. – Haris Khan Jan 20 '17 at 14:15
1

Hey Haris, you probably found your answer by now, but I'll answer for others. Docker keeps several copies of past services, but deletes them as you update. So, if you're history is full, the oldest copy will be deleted immediately and the most recently deleted copy will be added to your history. – Geige V Mar 04 '17 at 03:03
3

I needed to clear it right away so I set limit to 1. In next query, it listed only 'running' ones. – Hari Oct 23 '17 at 05:35
5

Even after removing all exited containers and setting task history limit to 1 I still see shutdown tasks. Are these instructions still viable or do they no longer work in recent versions of Docker? – Micah Zoltu Jan 16 '19 at 07:32

genmai · Answer 2 · 2020-07-14T21:19:22.463

yi92mgl7z8jb        web.2               nginx:latest        manager1            Running             Running 11 minutes ago
0cmzbd1oxwqr         \_ web.2           nginx:latest        manager1            Shutdown            Failed 11 minutes ago     "task: non-zero exit (255)"
moe7hex4qvmg         \_ web.2           nginx:latest        manager1            Shutdown            Shutdown 11 minutes ago
iyxs118uo67d         \_ web.2           nginx:latest        manager1            Shutdown            Shutdown 10 hours ago
v3uxafpxc4d3         \_ web.2           nginx:latest        manager1            Shutdown            Shutdown 11 minutes ago
6upsy8gvyrsn        web.5               nginx:latest        manager1            Running             Running 11 minutes ago
mlaxkfusunqe         \_ web.5           nginx:latest        manager1            Shutdown            Failed 11 minutes ago     "task: non-zero exit (255)"
bh3nkp05yd6r         \_ web.5           nginx:latest        manager1            Shutdown            Shutdown 11 minutes ago
lqedayxq7gr9         \_ web.5           nginx:latest        manager1            Shutdown            Shutdown 10 hours ago
xryxpfjsrdja         \_ web.5           nginx:latest        manager1            Shutdown            Shutdown 11 minutes ago

I did

docker swarm update --task-history-limit 2

docker-machine stop manager1

docker-machine start manager1

and

PS C:\> docker service ps web | Select-String "manager1"

3ogu1r0y6s6t        web.2               nginx:latest        manager1            Running             Running 3 minutes ago
0cmzbd1oxwqr         \_ web.2           nginx:latest        manager1            Shutdown            Failed 3 minutes ago      "task: non-zero exit (255)"
wbxr5hubftfa        web.5               nginx:latest        manager1            Running             Running 3 minutes ago
bh3nkp05yd6r         \_ web.5           nginx:latest        manager1            Shutdown            Shutdown 25 minutes ago

Thank you, Geige V

How can I remove shutdown Docker Service tasks after a rolling update?

2 Answers2