0

I have a python application that writes some data to AWS elastic cache cluster at a regular interval.

Here's an example script that simulates the functionality of my application. It also replicates the error that I have been facing.

from datetime import datetime
import redis
import time
import sys
import random

class Config:
    default_host = "localhost"
    master_host = "xxx.xx.0001.xxxx.cache.amazonaws.com"
    replica_host = "xxx.xx.0001.xxxx.cache.amazonaws.com"
    redis_db = 8
    socket_conn_timeout = 10
    request_delay_sec = 0.1

def get_redis_client():
    return redis.Redis(
        host=Config.master_host,
        db=Config.redis_db,
        socket_connect_timeout=Config.socket_conn_timeout,
    )

def get_random_key_value():
    val = time.time()
    key = "test_key_" + str(random.randint(0, 100))
    return key, val

r = get_redis_client()

r.flushdb()

flag = False

while True:
    try:
        if flag:
            print("beat:", time.time())
        r.set(*get_random_key_value())
        time.sleep(Config.request_delay_sec)
    except redis.RedisError as re:
        print(datetime.now(), "Error:", type(re), re)
        flag = True
        # sys.exit()
    except KeyboardInterrupt:
        print("Stopping loop execution")
        sys.exit()

Here are the environment details of my application

  • Python(v 3.7.0)
  • Redis-py(v 3.5.3)
  • AWS elastic cache(cluster mode disabled, 1 master node, 1 read replica)

When I scale my AWS elastic cache cluster vertically and the above script is running, I get the following error for few seconds while the cluster scaling-up is in process and then it goes away.

<class 'redis.exceptions.ReadOnlyError'> You can't write against a read only replica.

AWS docs also states that during vertical scaling process some inconsistencies may occur because of data syncing(https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/redis-cluster-vertical-scaling.html)

Has anyone faced any similar issue or can explain why this error occurs during the scale up process? how can it be fixed?

EDIT: I tried the same thing with a golang script and it works perfectly fine.

Akshit Sadana
  • 1
  • 2
  • Does this answer your question? https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Scaling.RedisReplGrps.html – gshpychka Dec 01 '21 at 17:27
  • It states that some inconsistency may occur, but I don't think the error I am getting is related to this – Akshit Sadana Dec 02 '21 at 08:01
  • "You will notice a brief interruption (a few seconds) of reads and writes from the old nodes because the connections to the old nodes will be disconnected." - this might be what's happening – gshpychka Dec 02 '21 at 09:03
  • It's working fine when using a golang based version of above script. – Akshit Sadana Dec 02 '21 at 15:45

0 Answers0