1

Cross-posted from k8s discuss.

Cluster information:

Kubernetes version: v1.17.1 Cloud being used: bare-metal Installation method: kubeadm Host OS: Ubuntu 18.04 LTS Server CNI and version: weave/flannel/kube-router/calico (latest releases of each) CRI and version: docker/containerd (19.03.5/1.2.10)

Problem:

I am attempting to bring up a ROS 2 installation on Kubernetes, ideally using multiple containers in a single pod. Under the hood, ROS 2 relies upon DDS for communication, which is based upon UDP multicast.

When I bring up a simple pod deployment with two containers in a producer-consumer configuration, the consumer rarely (if ever) receives a message from the producer. When I bring up two pods, each with a single container the same producer-consumer configuration, the consumer always receives the messages.

Surprises

Every once in a while, the consumer will start up and receive messages as expected.

Furthermore, if one logs into the consumer with kubectl exec -it ros2-1 -c consumer /bin/bash then runs /ros_entrypoint.sh ros2 run demo_nodes_cpp listener, messages are sometimes received from the producer in the single pod scenario.

Expected Behavior

Successful messages appear in the logs of the consumer container as:

[INFO] [1579805884.017171859] [listener]: I heard: [Hello World: 1]
[INFO] [1579805885.017168023] [listener]: I heard: [Hello World: 2]
[INFO] [1579805886.017025092] [listener]: I heard: [Hello World: 3]

Actual Behavior

No such log messages are observed from the consumer.

Steps to Reproduce:

Failure within same pod

  1. Bring up a kubernetes cluster

  2. Apply the following pod definition: ros2-1.yaml

    apiVersion: v1
kind: Pod
metadata:
  name: ros2-1
spec:
  containers:
  - name: producer
    image: osrf/ros2:nightly
    args: ["ros2", "run", "demo_nodes_cpp", "talker"]
  - name: consumer
    image: osrf/ros2:nightly
    args: ["ros2", "run", "demo_nodes_cpp", "listener"]
  restartPolicy: Never
  3. Watch for messages from the consumer with kubectl logs --follow ros2-1 consumer.

Success in different pods

  1. Bring up a kubernetes cluster

  2. Apply the following pod definition: ros2-2.yaml

    apiVersion: v1
kind: Pod
metadata:
  name: ros2-2-producer
spec:
  containers:
  - name: producer
    image: osrf/ros2:nightly
    args: ["ros2", "run", "demo_nodes_cpp", "talker"]
  restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
  name: ros2-2-consumer
spec:
  containers:
  - name: consumer
    image: osrf/ros2:nightly
    args: ["ros2", "run", "demo_nodes_cpp", "listener"]
  restartPolicy: Never
  3. Watch for messages from the consumer with kubectl logs --follow ros2-2-consumer.

Questions:

  1. What is causing a single pod deployment to fail, but multi pod deployment to succeed?
  2. I am unfamiliar with debugging networking issues within the Kubernetes environment, while fairly experienced on bare-metal. How should I go about investigating this issue under flannel, weave, or kube-router?
oursland
  • 11
  • 1
  • Is your loopback up? – suren Jan 23 '20 at 22:35
  • I tried it on 2 of my clusters and it works. The only difference now I can see between mine and your cluster is Ubuntu version. I will try to run it on your version later and I will let you know. – Matt Jan 24 '20 at 14:16
  • Try adding route in pod: `route add -net 224.0.0.0 netmask 240.0.0.0 dev lo` as I have found [here](https://stackoverflow.com/a/30982079/12201084) – Matt Jan 28 '20 at 13:48

1 Answers1

0

I believe this is answered in this ubuntu.com article:

Additionally, ROS 2 does not provide a method for managing ports used by RTPS. For example, a container can not change the standard RTPS discovery port of 7400, nor can a ROS 2 listener select a port other than its default. As a result, port usage can not be coordinated across multiple ROS 2 containers running in the same pod, and these containers generally will not be able to communicate.

vecf
  • 1