Route container traffic to separate non-container host via wireguard container

Question

My setup is basically two containers on one host (Host A), a prometheus container, and a wireguard container. And one a separate host (Host B) I have wireguard set up (not in a container).

On host A the containers are set up such that the wireguard container is creating a network and the prometheus container is on that network. The prometheus is also on other networks to talk to the other containers in it's stack, mainly cAdvisor and Grafana.

On host B I am running node_exporter and would like to have it report back to the prometheus instance, but I don't just want that done in the open.

Hence my plan was to have a wireguard tunnel between the two hosts and then having the prometheus container on host A and node_exporter on host B communicate that way.

My issue is that whilst host B is perfectly capable of reaching both the wireguard and the prometheus container (it can ping both), and whilst my wireguard container can reach host B, the prometheus container can not resolve the IP for host B. It simply says Get "http://10.10.10.20:9100/metrics": dial tcp 10.10.10.20:9100: connect: no route to host

Unfortunately I can not test ping from the prometheus container due to it running busybox and so ping always errors out askign if I'm root... But for all other use cases I can ping, i.e host B to both containers and WG container to host B.

Here are my configs.

Docker compose for the wireguard container and network:

services:
  server:
    image: procustodibus/wireguard:latest
    cap_add:
      - NET_ADMIN
    networks:
      wg-network:
        ipv4_address: 10.10.10.99
    ports:
      - "51822:51822/udp"
    volumes:
      - ./wg0.conf:/etc/wireguard/wg0.conf

networks:
  wg-network:
    name: wg-network
    ipam:
      config:
        - subnet: 10.10.10.0/24

The prometheus container is setup with, among others, the network wg-network and an assigned IP 10.10.10.10

with the following config

[Interface]
PrivateKey = ********
ListenPort = 51822
PreUp = iptables -t nat -A POSTROUTING -d 10.10.10.0/24 -j MASQUERADE

[Peer]
PublicKey = ********
Endpoint =  *******:51822
AllowedIPs = 10.10.10.20/32

on host B I have the following WG config

[Interface]
PrivateKey = *******
ListenPort = 51822

[Peer]
PublicKey =  *******
Endpoint =  *******:51822
AllowedIPs = 10.10.10.99/32, 10.10.10.10/32

On host B the wg0 device is set up, the ip is added to it (10.10.10.20/24) and the link is set to up. Following the standard WG setup from their website. On both hosts UFW is set up to accept UDP connections on port 51822.

Since host B can reach (ping) everything on host A (both containers on 10.10.10.10 and 10.10.10.99 respectively), and the WG container can reach host B on 10.10.10.20 I am assuming this is either something to do with me missing some UFW/Iptables thing i need to configure. Or me simply misunderstanding how the routing works between the prometheus and the WG container.

If you immediately think of some easier way to resolve this, I'm all ears. Although I would prefer not to have to set up certificates and TLS for the node_exporter instance on host B, if possible.

I am aware of network_mode: service:<container-name> but I can not use this because I need the prometheus container to also be on other networs (both the one for that stack to talk with grafana and cadvisor plus another one it is added to that allows it to talk to traefik) and setting network_mode means you can't also have a networks config for the service, this is simply a docker compose limitation.

Answer 1

The problem is that your Prometheus container doesn't know how to route to Host B (10.10.10.20). When it tries to connect, it sends the connection through the container's host (which also doesn't know how to route to 10.10.10.20).

There's two approaches you could use to solve this:

1. Use DNAT in the WireGuard container to forward connections from the Prometheus container to Host B.
2. Add a route to the Prometheus container for Host B.

1. Use DNAT in the WireGuard container to forward connections from the Prometheus container to Host B.

You can add a DNAT iptables rule to the WireGuard container (10.10.10.99) to forward connections made to its own TCP port 9100 (or some other convenient port) to Host B's (10.10.10.20) port 9100.

Add the following line to the WireGuard container's configuration:

PreUp = iptables -t nat -A PREROUTING -p tcp --dport 9100 -j DNAT --to-destination 10.10.10.20

Then change your Prometheus configuration to scrape 10.10.10.99:9100 instead of 10.10.10.20:9100 (and restart both containers).

2. Add a route to the Prometheus container for Host B.

Alternatively, you can adjust the routing within the Prometheus container itself to configure it to route to Host B (10.10.10.20) directly through your WireGuard container (10.10.10.99). If you do this, you can remove the MASQUERADE iptables rule from the WireGuard container's configuration.

You can do this as root on the container host if you know the Prometheus container's PID. You can lookup the container's PID if you know its Docker ID or name. For example, if the container's name is my-prometheus-container, you can lookup its PID with the following command:

$ sudo docker container inspect my-prometheus-container -f '{{.State.Pid}}'
12345

With the PID in hand, you can use the nsenter utility to run other programs on the container host within the container's namespace. If the container's PID is 12345, you can add a route within the container with the following command:

$ sudo nsenter -t 12345 -n ip route add 10.10.10.20/32 via 10.10.10.99

You'll have to do this every time you restart the Prometheus container.