Podman(1): How to route requests between the HOST and PODs attached to a podman network (rootless)

Question

Let me use a real example to aid this question.

Using rootless podman(1), I start a tiny Spark cluster within it's own Pod using the following script, which works well:

#! /usr/bin/bash

podman network create --subnet 192.168.10.0/24 --gateway 192.168.10.1 mynet

podman pod create \
    --name=spark \
    --network=mynet \
    --share net \
    --publish 7077:7077 \
    --publish 8080:8080

podman run \
    --detach \
    --pod=spark \
    --name=master01 \
    --hostname spark \
    --restart always \
    --env SPARK_MODE=master \
    --env SPARK_RPC_AUTHENTICATION_ENABLED=no \
    --env SPARK_RPC_ENCRYPTION_ENABLED=no \
    --env SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no \
    --env SPARK_SSL_ENABLED=no \
    --add-host spark:127.0.0.1 \
    --add-host worker01:127.0.0.1 \
    docker.io/bitnami/spark:latest

podman run \
    --detach \
    --pod=spark \
    --name=worker01 \
    --hostname worker01 \
    --restart always \
    --env SPARK_MODE=worker \
    --env SPARK_MASTER_URL=spark://spark:7077 \
    --env SPARK_WORKER_MEMORY=4G \
    --env SPARK_WORKER_CORES=8 \
    --env SPARK_RPC_AUTHENTICATION_ENABLED=no \
    --env SPARK_RPC_ENCRYPTION_ENABLED=no \
    --env SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no \
    --env SPARK_SSL_ENABLED=no \
    --add-host spark:127.0.0.1 \
    --add-host worker01:127.0.0.1 \
    docker.io/bitnami/spark:latest

Following is an image of the SparkUI.

(Note: This question uses Spark as an aid, but is not itself about Spark).

See lower-left corner of the browser image. Hovering the mouse over the only available Spark worker link (beneath Worker id), notice that the URL resolves to 192.168.10.2, which is a valid Pod IP-Address falling within the podman network created above.

However, clicking that URL will, of course, hang because that Pod IP-Address is not reachable from the HOST. With docker(1) (not podman(1)) there's a gateway mechanism to routes requests between guest containers and the Host.

How is this accomplished with podman(1)?

A few other points to help:

• This is an example Spark Pod. There are additional Pods that attach that same pod network: mynet
• I don't want to publish extra ports, because that will break anyway once you go deeper into the UI. More importantly, doing that defeats the purpose of Pods, which isolate conflicting ports from one another. Rather, I need to understand how to "gateway" these requests.
• Finally, I'm humbled because I thought I knew podman(1) quite well. LoL

Thank you in advance.

Answer 1

ANSWER:

In speaking with the podman(1) team over at GitHub, the scenario above (and similar) will always be problematic because rootless networking does not have privileges to configure bridge networking that could permit the port-forwarding needed.

So there are two alternatives:

1. Do the same thing above, but using rootful podman(1) (rootful containers). Basically, run things with sudo(1).
2. Use host mode networking, which places all containers into the same network namespace as the HOST (basically bare-metal naked) -- whether or not those containers were launched inside a Pod. Consequently, use of Pods with host mode no longer offers network isolation, but they still offer the convenience of treating containers within them as a group (for example start and stop them as a group). Also, the --publish option no longer applies because everything co-exists at the HOST (top) level; and so the notion of inner and outer ports don't exist.

A nice little tutorial for rootless container/host networking with podman(1) can be found here.