Python inside docker container, gracefully stop

Question

I'm running a very basic example of Python loop inside a Windows docker container, that I would like to gracefully stop.

The script is started this way in my dockerfile:

CMD [ "python.exe" , "./test.py"]

In the docker documentation it's said a SIGTERM signal is sent to the main command, so I'm trying to catch it this way:

import signal
import time
import logging, sys

class GracefulKiller:
  kill_now = False
  def __init__(self):
    signal.signal(signal.SIGINT, self.exit_gracefully)
    signal.signal(signal.SIGTERM, self.exit_gracefully)

  def exit_gracefully(self,signum, frame):
    self.kill_now = True

if __name__ == '__main__':
  logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
  killer = GracefulKiller()
  while True:
    time.sleep(1)
    logging.info("doing something in a loop ...")
    if killer.kill_now:
      break

  logging.info("End of the program. I was killed gracefully :)")

In theory the signal should be caught by the handler, the boolean should toggle and the loop should exit and display my very last log line. It doesn't, it's just stopping the whole thing at the moment the signal is emitted (or 2-3 seconds later rather)

C:\Users\Administrator\Documents\Projects\test>docker-compose up
Recreating test_1 ... done
Attaching to test_1
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
test_1  | INFO:root:doing something in a loop ...
Gracefully stopping... (press Ctrl+C again to force)
Stopping test_1   ... done

My last line log is never reached. Does anyone knows what's going on ? Is it a python specific issue, docker specific or Windows specific?

Also I tried to inspect the stopped container with docker logs, the last log isn't here either. Tried to add a sleep after it, same result.

Thanks,

Did you ever get this to work with `docker-compose`? – Gino Mempin Jun 15 '19 at 07:16 — Gino Mempin, Jun 15 '19 at 07:16

lys · Answer 1 · 2021-11-27T01:52:04.127

this seems to still be pervasive when using docker-compose up in current versions which i've been investigating on Raspbian all morning (and led me here).

However running your example with docker-compose 2.1.1 via docker-compose up with the following configs shows that the last line of your python code is actually called, you just can't see it:

docker-compose.yaml

services:
  grace_test:
    container_name: grace_test
    build: .

Dockerfile

FROM python:3.8-slim-buster

# setup WORKDIR
ADD . /grace_test
WORKDIR /grace_test
CMD ["python", "test.py"]

test.py

import signal
import time
import logging, sys

class GracefulKiller:
  kill_now = False
  def __init__(self):
    signal.signal(signal.SIGINT, self.exit_gracefully)
    signal.signal(signal.SIGTERM, self.exit_gracefully)

  def exit_gracefully(self,signum, frame):
    self.kill_now = True

if __name__ == '__main__':
  logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
  killer = GracefulKiller()
  while True:
    time.sleep(1)
    logging.info("doing something in a loop ...")
    if killer.kill_now:
      break

  logging.info("End of the program. I was killed gracefully :)")

Verify

Use docker-compose logs to inspect logs after Ctrl-C:

$ docker-compose logs
grace_test  | INFO:root:doing something in a loop ...
grace_test  | INFO:root:doing something in a loop ...
grace_test  | INFO:root:doing something in a loop ...
grace_test  | INFO:root:doing something in a loop ...
grace_test  | INFO:root:End of the program. I was killed gracefully :)

score 1 · Answer 2 · answered Dec 19 '18 at 13:59

1

Just catch KeyboardInterrupt and that's it.

if __name__ == '__main__':
  logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
  try:
    while True:
      time.sleep(1)
      logging.info("doing something in a loop ...")
  except KeyboardInterrupt as ex:
    print('goodbye!')

answered Dec 19 '18 at 13:59

grapes

8,185
1
19
31

The OP asks how to use operating system signals. – tgogos Dec 19 '18 at 14:10
1

@tgogos Where? Not in header, nor in text I found that OP needs OS signals. From the header and spirit of question I see, he needs the way to stop the container, no matter how. – grapes Dec 19 '18 at 14:12
1

I don't NEED signals, it's just that docker seems to be using it to gracefully stop a container: https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/ – Tigzy Dec 19 '18 at 14:19
6

@Tigzy your code seems to be working when I use `docker run ...`, but fails when I use `docker-compose up`. There are some issues on github mentioning this erratic behavior ([#3347](https://github.com/docker/compose/issues/3347), [#3317](https://github.com/docker/compose/issues/3317)). – tgogos Dec 19 '18 at 14:23
So that's a bug with docker-compose? Ok let me try with docker – Tigzy Dec 19 '18 at 14:32
@tgogos what do you mean working, you can see the ending logs? I tried with docker run here and still nothing. Are you on Windows? – Tigzy Dec 19 '18 at 14:41
No, I'm on Linux. – tgogos Dec 19 '18 at 14:44
Yeah that's why I suspect it's maybe a Windows issue. Will try on linux see what it does, but ultimately my production code will need to run on Windows, that's why... – Tigzy Dec 19 '18 at 14:58
Unfortunately I have no Windows to try, but could you try explicitly specifing stop signal type for docker like `docker kill --signal ...` – grapes Dec 19 '18 at 15:00
Yeah I did, same results with --signal=SIGTERM – Tigzy Dec 19 '18 at 15:03
Did you remove your signal handlers when tried with `KeyboardInterrupt` ? – grapes Dec 19 '18 at 15:11
Nope, I just added the try/except block around it. – Tigzy Dec 19 '18 at 15:18
Your handlers could shadow the interrupt signal – grapes Dec 19 '18 at 15:19
@grapes what do you mean by shadowing the interrupt signal? Can you illustrate? – Tigzy Dec 19 '18 at 15:30
2

Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/185496/discussion-between-grapes-and-tigzy). – grapes Dec 19 '18 at 15:36

Python inside docker container, gracefully stop

2 Answers2

docker-compose.yaml

Dockerfile

test.py

Verify

Linked