Redis Sentinel on Docker Gives Docker Network IP Addresses

Question

My objective is to set up 3x Redis Server and 3x Redis Sentinel on a single Docker VM using Docker compose and expose each of the Redis Servers and Sentinels to the local network.

The static IP of my Docker host is 192.168.2.90.

I've given the Redis Servers ports numbered 6379, 6380, 6381 and exposed those ports through Docker.

My local network is 192.168.2.0/24

My Docker machine's internal network is 172.16.0.0/12.

Everything works great within the Docker containers themselves. The problem comes when I try to connect to Redis using a different machine on my local network.

My python test script successfully connects to Redis Sentinel. The problem is when it discovers the master and slaves, the addresses are all on the 172.16.0.0/12 subnet.

Redis master: 
('172.18.3.1', 6379)

Redis slaves: 
[('172.18.3.3', 6381), ('192.168.2.90', 6379),('172.18.3.2', 6380)]

When I telnet into the master and run INFO, likewise it gives me the 172.16.0.0/12 addresses.

role:master
connected_slaves:2 
slave0:ip=172.18.3.3,port=6381,state=online,offset=252692195,lag=0
slave1:ip=172.18.3.2,port=6380,state=online,offset=252692195,lag=0

I cannot figure out how to get Redis Server and Redis Sentinel to report the 192.168.2.0/24 subnet.

I've defined my Redis Server containers as follows:

  redis-a-1:
    container_name: redis-a-1
    hostname: redis-a-1
    image: redis
    command: "redis-server --port 6379 --bind-source-addr 192.168.2.90"    
    environment:
      - REDIS_HOST=192.168.2.90
    ports:
    - "6379:6379"
    restart: unless-stopped
    networks:
      blue-green-network:
        ipv4_address: 172.18.3.1

  redis-a-2:
    container_name: redis-a-2
    hostname: redis-a-2
    image: redis
    command: "redis-server --port 6380 --bind-source-addr 192.168.2.90 --slaveof 192.168.2.90 6379"        
    environment:
      - REDIS_HOST=192.168.2.90
    ports:
    - "6380:6380"
    restart: unless-stopped
    networks:
      blue-green-network:
        ipv4_address: 172.18.3.2

  redis-a-3:
    container_name: redis-a-3
    hostname: redis-a-3
    image: redis
    command: "redis-server --port 6381 --bind-source-addr 192.168.2.90 --slaveof 192.168.2.90 6379"        
    environment:
      - REDIS_HOST=192.168.2.90
    ports:
    - "6381:6381"
    restart: unless-stopped
    networks:
      blue-green-network:
        ipv4_address: 172.18.3.3

I've tried a bunch of things to get each of the Redis Servers to report their IP addresses to each other as the Docker host's IP address but with no success.

Any thoughts on how I can do this or am I on a fool's erand?

Many thanks.

Note: I am aware that the machine itself is a single point of failure.

score 1 · Answer 1 · answered May 19 '23 at 20:56

What you have done by exposing the ports of the Redis servers is to create a mapping from the host machine's IP address to the internal Docker IP addresses. However, this does not change the IP addresses that the Redis servers see and use to communicate with each other. They still see their own IP addresses as being in the 172.16.0.0/12 subnet.

Docker, by default, isolates containers from the host machine and other containers, and it provides each container with its own network stack. Containers are assigned IP addresses from an internal Docker subnet (in your case, 172.16.0.0/12), and these addresses are not usually accessible from outside the Docker host.

You are using Redis Sentinel (which should help manage Redis server topologies, providing services like monitoring, notifications, automatic failover and configurations), and when it queries the Redis servers, they report their internal Docker IP addresses: that is what you're seeing.

Reading "IP Addresses and DNS names", you could use "announce-ip": that would tell Redis to announce a different IP address to clients and other Redis servers.

The --announce-ip option can be used to tell a Redis server to announce a different IP address to clients and other Redis servers, which can help when Redis is running inside a Docker container and needs to communicate with outside entities.

Your issue is with the Redis Sentinel, which is having trouble discovering the Redis servers and other Sentinel instances due to Docker's port remapping. According to the Redis documentation, you should use the sentinel announce-ip <ip> and sentinel announce-port <port> directives in your Sentinel configuration to solve this issue1.

redis-sentinel-1:
  container_name: redis-sentinel-1
  hostname: redis-sentinel-1
  image: redis
  command: "redis-server /etc/redis/sentinel.conf --sentinel --announce-ip 192.168.2.90 --announce-port 26379"
  environment:
    - REDIS_HOST=192.168.2.90
  ports:
  - "26379:26379"
  restart: unless-stopped
  networks:
    blue-green-network:
      ipv4_address: 172.18.3.4

This sets up a Redis Sentinel using the Redis image and runs the Sentinel service with the --announce-ip and --announce-port options.
Replace 26379 with the appropriate port number for your Sentinel instance. Also, note that you need to have a sentinel configuration file (in this case, /etc/redis/sentinel.conf) in your Docker container for this to work.

Repeat this for redis-a-2 and redis-a-3, and hopefully, this should solve your problem.

"Docker in host networking mode" would be a simpler solution, but only if it is compatible with your setup and security requirements.

In host networking mode, the Docker container shares the network stack with the Docker host itself, meaning it does not perform any Network Address Translation (NAT) or port mapping. This can alleviate the issues you're experiencing with Redis Sentinel's auto-discovery and failover mechanisms.

But obviously, this is less secure (containers have less network isolation in host mode, because they share the host's network namespace and have access to all its network interfaces), and can include port conflict with application on your host.

Redis Sentinel on Docker Gives Docker Network IP Addresses

1 Answers1