Issue while connecting to redis master node running in sentinel mode in docker conatiners

Question

I am running redis in docker containers and I am using redis sentinel mode. I have setup the following configuration -

3 redis sentinels nodes
1 redis master node
2 redis slave nodes

I am running all these in my local machine. So in total 6 docker containers running through docker-compose in bridge networking mode. All conatiners has port mappings to outside.

All containers can access each other as they are in the same bridge docker network created during running of docker-compose up.

I have created a Java client using Redisson library to access redis. Configured the client to use redis-sentinel mode as follows -

Config config = new Config();
config.useSentinelServers()
       .setMasterName("redis-master")
       .addSentinelAddress("redis://127.0.0.1:26379")
       .addSentinelAddress("redis://127.0.0.1:26380")
       .addSentinelAddress("redis://127.0.0.1:26381")

RedissonClient client = Redisson.create(config);

This is where I am facing the issue. Whenever I try to run some commands on redis through this client, the request goes through sentinel nodes which gives me current redis master node address. But my java client cannot communicate to redis master directly as the ip returned by sentinel node is the internel docker network ip for master node which is not accessible outside docker network and it fails with similar exception as below -

Exception in thread "main" org.redisson.client.RedisConnectionException: Unable to connect to Redis server: 172.21.0.2/172.21.0.2:6379
    at org.redisson.connection.pool.ConnectionPool$2$1.operationComplete(ConnectionPool.java:161)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)

How to fix this issue? Do I need to run it in some different network mode? or some way to translate this internal docker ip to the actual ip of machine running docker conatiners?

Answer 1

I ran into this issue today as well, attempting to get this set up for a test instance. My initial compose file was based on https://blog.alexseifert.com/2016/11/14/using-redis-sentinel-with-docker-compose/ with modifications to suit my needs. I was able to find a workaround by 1) binding the ports to my host machine, 2) setting the depends_on flag in docker-compose, and 3) setting my sentinel.conf to point to the hostname that my master was running on: https://docs.docker.com/compose/startup-order/

It's a little hard to explain, but I'll try my best: Your master/replica will look something like this in docker-compose:

  redis-master:
    image: redis:5.0.4-alpine
    volumes:
      - <mounted-data-directory>
      - "<local-master-config-directory>/redis.conf:/usr/local/etc/redis/redis.conf"
    ports:
      - "6379:6379"
    command:
      - redis-server
      - /usr/local/etc/redis/redis.conf

  redis-replica:
    image: redis:5.0.4-alpine
    links:
      - redis-master
    volumes:
      - <mounted-data-directory>
      - "<local-replica-config-directory>:/usr/local/etc/redis/redis.conf"
    ports:
      - "6380:6380"
    depends_on:
      - redis-master
    command:
      - redis-server
      - /usr/local/etc/redis/redis.conf
      - --slaveof redis-master 6379

For my sentinels I gave each Dockerfile and a sentinel.conf (each having different ports): Dockerfile:

FROM redis:5.0.4-alpine
RUN mkdir -p /redis
WORKDIR /redis
COPY sentinel.conf .
RUN chown redis:redis /redis/*
ENTRYPOINT ["redis-server", "/redis/sentinel.conf", "--sentinel"]

sentinel.conf

port 26379
dir /tmp
bind 0.0.0.0
sentinel monitor mymaster <hostname> 6379 2
sentinel down-after-milliseconds mymaster 1000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 10000

It's worth noting that I attempted to do this with 127.0.0.1 and localhost and I don't think either worked, so I had the hostname set on the machine I ran this on. I was kind of trying any and everything at that point.

Each sentinel (I had three) had a separate entry referencing their build contexts and mapping the port in sentinel.conf to the local port. So in docker-compose my sentinels looked like this:

 # Instance 1
  redis-sentinel:
    build:
      context: <path-to-context>
    links:
      - redis-master
    ports:
      - "26379:26379"
    depends_on:
      - redis-replica

What I did was definitely a hack and I wouldn't do it in production. I'm pretty sure there's a much better networking solution for docker, I just didn't want to go too far down the rabbit hole for what I needed to test. Hope this helps.