I am facing a strange issue with a Redis HA based on Sentinel.
After a period of time, Redis is closing the connection of all the Sentinels (not at the same time) with the following log entry:
14766:M 25 May 2023 14:13:14.156 # Client id=10 addr=10.23.163.168:26489 laddr=10.23.162.168:6370 fd=14 name=sentinel-182d3e93-pubsub age=676494 idle=115562 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16327 oll=410 omem=8406640 tot-mem=8427144 events=rw cmd=subscribe user=default redir=-1 closed for overcoming of output buffer limits.
After this, Sentinel doesn't retry to connect and connection towards the Redis node is lost. I understood that I can avoid this by increasing the client-output-buffer-limit bellow the configuration that I am currently using:
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
But before moving forward with this I cannot find any answer to the following questions:
a) Why Sentinel doesn't try to reconnect. Is there any configuration/way to control this? b) How much should I increase the client-output-buffer-limit? Is there any tuning/guidance around this?
