Naming current Redis master (run in Docker) in Python

Question

How to make Python failover to Redis slave if master is down?

With current configuration, Sentinels elect a new master, but Python writes stop.

I assume I should not use redis-master as is in docker-compose.yml file; what are the alternatives?

In networking mode, I couldn't get Sentinels to recognize each other.

docker-compose.yml

version: '3'

services:

  redis-master:
    container_name: redis-master
    image: redis:latest
    command: redis-server --port 6379
    ports:
      - "6379:6379"
    volumes:
      - .:/app

  redis-slave:
    container_name: redis-slave
    image: redis:latest
    command: redis-server --slaveof redis-master 6379 --protected-mode no
    volumes:
       - .:/app

  sentinel-1:
    container_name: sentinel-1
    build: sentinel
    environment:
      - SENTINEL_DOWN_AFTER=5000
      - SENTINEL_FAILOVER=5000

  sentinel-2:
    container_name: sentinel-2
    build: sentinel
    environment:
      - SENTINEL_DOWN_AFTER=5000
      - SENTINEL_FAILOVER=5000

  sentinel-3:
    container_name: sentinel-3
    build: sentinel
    environment:
      - SENTINEL_DOWN_AFTER=5000
      - SENTINEL_FAILOVER=5000

  app:
    container_name: python-app
    image: pyredis
    command: python app.py

Python app:

import redis
import random
import time

r = redis.StrictRedis(host="redis-master", port=6379, db=0)

for i in range(0, 1000):
    timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
    num = random.randint(1,101)
    r.set(timestamp, num)
    time.sleep(2)

Thank you, any input is appreciated.

C.Nivs · Accepted Answer · 2019-09-05T18:35:20.483

You'll need a function to re-define what your r value is:

import redis
from redis.exceptions import ConnectionError

def get_connection(host):
    global r
    other_host = "redis-slave" if "master" in host else "redis-master"

    try:
        r = redis.StrictRedis(host=host, port=6379, db=0)
    except ConnectionError:
        # connection against host failed, try other_host
        host = other_host
        r = redis.StrictRedis(host=host, port=6379, db=0)
    return host


host = get_connection('redis-master')

for i in range(0, 1000):
    timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
    num = random.randint(1,101)
    try:
        r.set(timestamp, num)
    # handles explicitly redis.exceptions.ConnectionError
    except ConnectionError: 
        host = get_connection(host)
        r.set(timestamp, num)
    time.sleep(2)

In the case of master going down, that function will be called and the connection will be switched to the slave, and vice versa. If both are down, the Exception caught by the outer except will be raised and the program will crash

Edit

This could be a really interesting use-case for mutable defaults which normally is a BIG NO-NO. Note, I would still not implement this in any production code, just showing the side-effect here

A deque supports in-place rotation, so you can use that to switch host and other_host like so:

from collections import deque

d = deque(('redis-master', 'redis-slave'), maxlen=2)

d
deque(['redis-master', 'redis-slave'], maxlen=2)

d.rotate()
d
deque(['redis-slave', 'redis-master'], maxlen=2)

Now, you can use that as your cache and just swap the order

def get_connection(d=deque(('redis-master', 'redis-slave'), maxlen=2)):
    global r
    host, other_host = d # unpacks the two values

    try:
        r = redis.StrictRedis(host=host, port=6379, db=0)
    except ConnectionError:
        r = redis.StrictRedis(host=other_host, port=6379, db=0)
        d.rotate() # changes the order of the hosts

To show how this works:

def get_connection(d=deque(('redis-master', 'redis-slave'), maxlen=2)):
    print(d)
    try:
        raise ValueError # as a test
    except ValueError:
        print("Caught error")
        d.rotate()
        print(d)


get_connection()
deque(['redis-master', 'redis-slave'], maxlen=2)
Caught error
deque(['redis-slave', 'redis-master'], maxlen=2)

get_connection()
deque(['redis-slave', 'redis-master'], maxlen=2)
Caught error
deque(['redis-master', 'redis-slave'], maxlen=2)

Now your external program doesn't need to know about host, it would just have to call the retry function as necessary:

for i in range(0, 1000):
    timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
    num = random.randint(1,101)
    try:
        r.set(timestamp, num)
    # handles explicitly redis.exceptions.ConnectionError
    except ConnectionError: 
        get_connection() # host doesn't need to be returned anymore
        r.set(timestamp, num)
    time.sleep(2)

Thank you! Do you perhaps think Redis will raise an error here? Such as ConnectionError? (https://redis-py.readthedocs.io/en/latest/_modules/redis/exceptions.html) — , Aug 29 '19 at 17:33

Naming current Redis master (run in Docker) in Python

1 Answers1

Edit