2

We are using Jenkins and Docker in combination.. we have set up Jenkins like master/slave model, and containers are spun up in the slave agents. Sometimes due to bug in jenkins docker plugin or for some unknown reasons, containers are left dangling.

Killing them all takes time, about 5 seconds per container process and we have about 15000 of them. Will take ~24hrs to finish running the cleanup job. How can I remove the containers bunch of them at once? or effectively so that it takes less time?

  1. Will uninstalling docker client, remove the containers?
  2. Is there a volume where these containers process kept, could be removed (bad idea)
  3. Any threading/parallelism to remove them faster? I am going to run a cron job weekly to patch these bugs, but right now I dont have whole day to get these removed.
Krish
  • 467
  • 1
  • 6
  • 16

3 Answers3

3

Try this:

  1. Uninstall docker-engine
  2. Reboot host
  3. rm /var/lib/docker

Rebooting effectively stops all of the containers and uninstalling docker prevents them from coming back upon reboot. (in case they have restart=always set)

Michael
  • 8,229
  • 4
  • 17
  • 14
3

If you are interesting in only killing the processes as they are not exiting properly (my assessment of what you mean--correct me if I'm wrong), there is a way to walk the running container processes and kill them using the Pid information from the container's metadata. As it appears you don't necessarily care about clean process shutdown at this point (which is why docker kill is taking so long per container--the container may not respond to the right signals and therefore the engine waits patiently, and then kills the process), then a kill -9 is a much more swift and drastic way to end these containers and clean up.

A quick test using the latest docker release shows I can kill ~100 containers in 11.5 seconds on a relatively modern laptop:

$ time docker ps --no-trunc --format '{{.ID}}' | xargs -n 1 docker inspect --format '{{.State.Pid}}' $1 | xargs -n 1 sudo kill -9

real    0m11.584s
user    0m2.844s
sys     0m0.436s

A clear explanation of what's happening:

  1. I'm asking the docker engine for an "full container ID only" list of all running containers (the docker ps)
  2. I'm passing that through docker inspect one by one, asking to output only the process ID (.State.Pid), which
  3. I then pass to the kill -9 to have the system directly kill the container process; much quicker than waiting for the engine to do so.

Again, this is not recommended for general use as it does not allow for standard (clean) exit processing for the containerized process, but in your case it sounds like that is not important criteria.

If there is leftover container metadata for these exited containers you can clean that out by using:

docker rm $(docker ps -q -a --filter status=exited)

This will remove all exited containers from the engine's metadata store (the /var/lib/docker content) and should be relatively quick per container.

Phil E
  • 1,840
  • 14
  • 19
1

So,

docker kill $(docker ps -a -q)

isn't what you need?

EDIT: obviously it isn't. My next take then:

A) somehow create a list of all containers that you want to stop.

B) Partition that list (maybe by just slicing it into n parts).

C) Kick of n jobs in parallel, each one working one of those list-slices.

D) Hope that "docker" is robust enough to handle n processes sending n kill requests in sequence in parallel.

E) If that really works: maybe start experimenting to determine the optimum setting for n.

GhostCat
  • 137,827
  • 25
  • 176
  • 248
  • Thanks GhostCat, thats the command I am using to kill them all, but it kills one at a time, sequentially and takes about 5 secs per process to be killed. So imagine for 15000 of them – Krish Sep 06 '16 at 15:51
  • Let me try that, I appreciate your thoughtful answer ! – Krish Sep 06 '16 at 15:56
  • You are welcome. Just for the record; added another final step. – GhostCat Sep 06 '16 at 15:59
  • thanks for the beautiful thought, unfortunately docker client is not just compatible with doing this in parallel ... i tired all different ways of parallelism nothing worked. I even wrote a python script which takes the input of those containers, slice them in chunks and spin off into 100 threads. but no luck – Krish Sep 06 '16 at 18:31
  • Glad to see that you got another working suggestion! – GhostCat Sep 06 '16 at 18:40