How to reconnect Redis cluster nodes?

Question

I have a Redis cluster of 6 nodes, running in my Kubernetes cluster as a stateful set. Since it was for testing and not yet on production, all of the Redis nodes were on the same machine. Of course, the machine failed, and all of Redis' nodes crashed immediately. When the machine was back alive the pods were recreated and was given different cluster ips, therefore they couldn't re-connect with each other.

I need to find a solution for a disaster case such as this. Let's assume all the nodes were reassigned with different ips, how can I configure the nodes to get to other ips?

The slaves are easy to reset with the CLUSTER RESET command, but the masters contain slots and data that shouldn't be deleted.

Should I manually re-write nodes.conf? I'm afraid this will make it even worse? I there a known method to deal with it?

Thanks!

score 2 · Accepted Answer · answered Feb 15 '19 at 14:13

Found a solution:

The first step is to change the current pod ip in nodes.conf when the pod is starting. You can achieve that with this script

#!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${CLUSTER_CONFIG}
    fi
    exec "$@"

You should start any pod by calling this script and passing to it the original redis-server start command.

Now every pod in the cluster has the correct IP of itself set.

Make sure the cluster's pods are stable and don't crash.
Manually edit nodes.conf in one of the pods. Set the correct IPs instead of the deprecated ones.
Restart the pod you've edit with redis-cli shutdown. Kubernetes will set up a new pod for it. The new pod's IP will be set by the script I added above.

score -1 · Answer 2 · answered Feb 15 '19 at 16:52

In my opinion you shouldn't rely on Pods' internal IP address at all, when referencing your Redis cluster in any place within your application. Pods are mortal, this means they are designed to crash. So when node dies, they are destroyed too. When node resurrects, PODs are recreated with new IP addresses.

The proper way to target your PODs would be via their DNS names (as explained here), if you created your Redis cluster as a Stateful application.

How to reconnect Redis cluster nodes?

2 Answers2