How to gracefully stop a Dockerized Python ROS2 node when run with docker-compose up?

Question

I have a Python-based ROS2 node running inside a Docker container and I am trying to handle the graceful shutdown of the node by capturing the SIGTERM/SIGINT signals and/or by catching the KeyboardInterrupt exception.

The problem is when I run the node in a container using docker-compose. I cannot seem to catch the "moment" when the container is being stopped/killed. I've explicitly added the STOPSIGNAL in the Dockerfile and the stop_signal in the docker-compose file.

Here is a sample of the node code:

import signal
import sys
import rclpy

def stop_node(*args):
    print("Stopping node..")
    rclpy.shutdown()
    return True

def main():
    rclpy.init(args=sys.argv)
    print("Creating node..")
    node = rclpy.create_node("mynode")
    print("Running node..")
    while rclpy.ok():
        rclpy.spin_once(node)

if __name__ == '__main__':
    try:
        signal.signal(signal.SIGINT, stop_node)
        signal.signal(signal.SIGTERM, stop_node)
        main()
    except:
        stop_node()

Here is a sample Dockerfile to re-create the image:

FROM osrf/ros2:nightly

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
RUN apt-get update && \
    apt-get install -y vim
WORKDIR /nodes
COPY mynode.py .
ADD run-node.sh /run-node.sh
RUN chmod +x /run-node.sh
STOPSIGNAL SIGTERM

Here is the sample docker-compose.yml:

version: '3'

services:
  mynode:
    container_name: mynode-container
    image: mynode
    entrypoint: /bin/bash -c "/run-node.sh"
    privileged: true
    stdin_open: false
    tty: true
    stop_signal: SIGTERM

Here is the run-node.sh script:

source /opt/ros/$ROS_DISTRO/setup.bash
python3 /nodes/mynode.py

When I manually run the node inside the container (using python3 mynode.py or by /run-node.sh) or when I do docker run -it mynode /bin/bash -c "/run-node.sh", I get the "Stopping node.." message. But when I do docker-compose up, I never see that message when I stop the container, by Ctrl+C or by docker-compose down.

$ docker-compose up
Creating network "ros-node_default" with the default driver
Creating mynode-container ... done
Attaching to mynode-container
mynode-container | Creating node..
mynode-container | Running node..

^CGracefully stopping... (press Ctrl+C again to force)
Stopping mynode-container ... done
$

I've tried:

• moving the calls to signal.signal
• using atexit instead of signal
• using docker stop and docker kill --signal

I've also checked this Python inside docker container, gracefully stop question but there's no clear solution there, and I'm not sure if using ROS/rclpy makes my setup different (also, my host machine is Ubuntu 18.04, while that user was on Windows).

Is it possible to catch the stopping of the container in my stop_node method?

Answer 1

When your docker-compose.yml file says:

entrypoint: /bin/bash -c "/run-node.sh"

Since that's a bare string, Docker wraps it in a /bin/sh -c wrapper. So your container's main process is something like

/bin/sh -c '/bin/bash -c "/run-node.sh"'

In turn, the bash script stays running. It launches a Python script, and stays running as its parent until that script exits. (The two levels of sh -c wrappers may or may not stay running.)

The important part here is that this wrapper shell, not your script, is the main container process that receives signals, and (it turns out) won't receive SIGTERM unless it's explicitly coded to.

The most important restructuring to do here is to have your wrapper script exec the Python script. That causes it to replace the wrapper, so it becomes the main process and receives signals. If nothing else changing the last line to

exec python3 /nodes/mynode.py

will likely help.

---

I would go a little further here and make sure as much of this code is built into your Docker image, and try to minimize the number of explicit shell wrappers. "Do some initialization, then exec something" is an extremely common Docker pattern, and you can write this script and make it your image's entrypoint:

#!/bin/sh
# Do the setup
# ("." is the same as "source", but standard)
. "/opt/ros/$ROS_DISTRO/setup.bash"
# Run the main CMD
exec "$@"

Similarly, your main script should start with a "shebang" line like

#!/usr/bin/env python3
import ...

Your Dockerfile already contains the setup to be able to run the wrapper directly, you may need a similar RUN chmod line for the main script. But then you can add

ENTRYPOINT ["/run-node.sh"]
CMD ["/nodes/my-node.py"]

Since both scripts are executable and have the "shebang" lines you can run them directly. Using the JSON syntax keeps Docker from adding an additional shell wrapper. Since your entrypoint script will now run whatever the command is, it's easy to change that separately. For example, if you want an interactive shell that's done the environment variable setup to try to debug your container startup, you can override just the command part

docker run --rm -it mynode sh