6

I have a docker-compose file that creates 3 Hello World applications and uses nginx to load balance traffic across the different containers.

The docker-compose code is as follows:

version: '3.2'
services:
  backend1:
      image: rafaelmarques7/hello-node:latest
      restart: always
  backend2:
      image: rafaelmarques7/hello-node:latest
      restart: always
  backend3:
      image: rafaelmarques7/hello-node:latest
      restart: always
  loadbalancer:
      image: nginx:latest      
      restart: always
      links:
          - backend1
          - backend2
          - backend3
      ports:
        - '80:80'
      volumes: 
        - ./container-balancer/nginx.conf:/etc/nginx/nginx.conf:ro

I would like to verify that the restart: always policy actually works.

The approach I tried is as follows:

  • First, I run my application with docker-compose up;
  • I identify the containers IDs with docker container ps;
  • I kill/stop one of the containers with docker stop ID_Container or docker kill ID_Container.

I was expecting that after the 3rd step (stop/kill the container. this makes it exist with code 137), the restart policy would kick in and create a new container again.

However, this does not happen. I have read that this is intentional, as to have a way to be able to manually stop containers that have a restart policy.

Despite this, I would like to know how I can kill a container in such a way that it triggers the restart policy so that I can actually verify that it is working.

Thank you for your help.

Rafael Marques
  • 1,335
  • 4
  • 22
  • 35

2 Answers2

5

If you run ps on the host you will be able to see the actual processes in all of your Docker containers. Once you find a container's main process's process ID, you can sudo kill it (you will have to be root). That will look more like a "crash", especially if you kill -13 to send SIGSEGV.

It is very occasionally useful for validation scenarios like this to have an endpoint that crashes your application that you can enable in test builds and some other similar silly things. Just make sure you do have a gate so that those endpoints don't exist in production builds. (In old-school C, an #ifdef TEST would do the job; some languages have equivalents but many don't.)

David Maze
  • 130,717
  • 29
  • 175
  • 215
  • I can confirm this work. Took me a while to understand it because I was unsure where to run the `ps` and `kill` commands (you should run it in your machine, and not inside the docker container.). Its also very tricky to find out which process are the docker ones. But in the end, I was able to kill the process and see it being restarted. Thank you :) – Rafael Marques Jan 31 '19 at 16:19
  • 1
    From the docker host, try `ps aux | grep docker`. The one you want matches your entry point. – sqqqrly Apr 16 '21 at 14:01
5

You can docker exec into the running container and kill processes. If your entrypoint process (pid 1) starts a sub process, find it and kill it

docker exec -it backend3 /bin/sh
ps -ef

Find the process that pid 1 is its parent and kill -9 it.

If your entrypoint in the only process (pid 1), it cannot be killed by the kill command. Consider replacing your entrypoint with a script that calls your actual process, which will allow you to use the idea I suggest above.

This should simulate a crashing container and should kick the restart process.

NOTES:

  1. See explanation in https://unix.stackexchange.com/questions/457649/unable-to-kill-process-with-pid-1-in-docker-container
  2. See why not run NodeJS as pid 1 in https://www.elastic.io/nodejs-as-pid-1-under-docker-images/
Eldad Assis
  • 10,464
  • 11
  • 52
  • 78
  • I run the first command and got inside the container. There, I run `ps` to see the processed, and i kill the main process with the command you provided (`kill -9 1`). However ,this has literally no effect. Am i douing something wrong? This question also suggests that this does not workhttps://stackoverflow.com/questions/46910412/docker-kill-process-inside-container – Rafael Marques Jan 31 '19 at 16:15
  • Yes - apparently `kill` to process id 1 is not possible. If pid 1 is calling another process - kill that process. If not, consider changing it so pid 1 is a script that runs the main command, so you can kill the spawned process and get that done. – Eldad Assis Jan 31 '19 at 19:03
  • Note the explanation in https://unix.stackexchange.com/questions/457649/unable-to-kill-process-with-pid-1-in-docker-container – Eldad Assis Jan 31 '19 at 19:16
  • 1
    uh, I appreciate the update, the resources you shared are very interesting, thank you! – Rafael Marques Feb 01 '19 at 09:44