How to run a redis cluster on a docker cluster?

Question

Context

I am trying to setup a redis cluster so that it runs on top off a docker cluster, to achieve maximum auto-healing.

More precisely, I have a docker compose file, which defines a service that has 3 replicas. Each service replica has a redis-server running on. Then I have a program inside each replica that listens to changes on the docker cluster and that starts the cluster when conditions are met (each 3 redis-servers know each other).

Setting up the redis cluster works has expected, the cluster is formed and all the redis-servers communicate well, but the communication between redis-servers is inside the docker cluster.

The Problem

When I try to communicate from outside the docker cluster, because of the ingress mode I am able to talk to a redis-server, however when I try to add info (eg: set foo bar) and the client is moved to another redis-server the communication hangs and eventually times out.

Code

This is the docker-compose file.

version: "3.3"
services:
    redis-cluster:
      image: redis-srv-instance
      volumes:
       - /var/run/:/var/run
      deploy:
        mode: replicated
        #endpoint_mode: dnsrr
        replicas: 3
        resources:
           limits:
             cpus: '0.5'
             memory: 512M
      ports:
        - target: 6379
          published: 30000
          protocol: tcp
          mode: ingress

The flux of commands that show the problem.

Client

~ ./redis-cli -c -p 30000
127.0.0.1:30000>

Redis-server

OK
1506533095.032738 [0 10.255.0.2:59700] "COMMAND"
1506533098.335858 [0 10.255.0.2:59700] "info"

Client

127.0.0.1:30000> set ghb fki
OK

Redis-server

1506533566.481334 [0 10.255.0.2:59718] "COMMAND"
1506533571.315238 [0 10.255.0.2:59718] "set" "ghb" "fki"

Client

127.0.0.1:30000> set rte fgh
-> Redirected to slot [3830] located at 10.0.0.3:6379
Could not connect to Redis at 10.0.0.3:6379: Operation timed out
Could not connect to Redis at 10.0.0.3:6379: Operation timed out
(150.31s)
not connected>

Any ideas? I have also tried making my one proxy/load balancer but didn't work.

Thank you! Have a nice day.

score 1 · Answer 1 · answered Sep 27 '17 at 18:01

For this use case, sentinel might help. Redis on its own is not capably of high availability. Sentinel on the other side is a distributed system which can do the following for you:

Route the ingress trafic to the current Redis master.
Elect a new Redis master should the current one fail.

While I have previously done research on this topic, I have not yet managed to pull to getter a working example.

score 0 · Answer 2 · answered Sep 27 '17 at 19:44

redis-cli would get the redis server ip inside the ingress network, and try to access the remote redis server by that ip directly. That is why redis-cli shows Redirected to slot [3830] located at 10.0.0.3:6379. But this internal 10.0.0.3 is not accessible to redis-cli.

One solution is to run another proxy service which attaches to the same network with redis cluster. The application sends all requests to that proxy service, and the proxy service talks with redis cluster.

Or you could create 3 swarm services that uses the bridge network and exposes the redis port to node. Your internal program needs to change accordingly.

How to run a redis cluster on a docker cluster?

2 Answers2