I am getting strange errors for my redis-sentinel. Anyone has any idea what is this?
2022-12-14T23:00:00.000+0000
The timeout was reached before the message could be written to the output buffer, and it was not sent, command=PING, timeout: 5000, inst: 0, qu: 0, qs: 0, aw: False, bw: CheckingForTimeout, rs: NotStarted, ws: Initializing, in: 0, last-in: 0, cur-in: 0, sync-ops: 1, async-ops: 8, serverEndpoint: 10.0.14.13:6379, conn-sec: n/a, mc: 1/1/0, mgr: 10 of 10 available, clientName: boydn-api-frontend(SE.Redis-v2.6.86.49666), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=5,Free=32762,Min=8,Max=32767), POOL: (Threads=16,QueuedItems=1,CompletedItems=5109581), v: 2.6.86.49666 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
latency_percentiles_usec_echo:p50=0.001,p99=1.003,p99.9=8.031 latency_percentiles_usec_set:p50=6.015,p99=17.023,p99.9=54.015 latency_percentiles_usec_ping:p50=1.003,p99=3.007,p99.9=18.047 latency_percentiles_usec_publish:p50=10.047,p99=28.031,p99.9=86.015 latency_percentiles_usec_del:p50=4.015,p99=20.095,p99.9=96.255
Cluster
cluster_enabled:0
Keyspace
db0:keys=4,expires=4,avg_ttl=0 ------ CLIENT LIST OUTPUT ------ id=218210 addr=10.0.14.43:38498 laddr=10.0.14.13:6379 fd=11 name=boydn-api-backend(SE.Redis-v2.6.86.49666) age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=info user=default redir=-1 resp=2 id=218211 addr=10.0.14.43:38512 laddr=10.0.14.13:6379 fd=12 name=boydn-api-backend(SE.Redis-v2.6.86.49666) age=1 idle=1 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22328 events=r cmd=subscribe user=default redir=-1 resp=2 id=218212 addr=10.0.14.47:51232 laddr=10.0.14.13:6379 fd=13 name=boydn-api-frontend(SE.Redis-v2.6.86.49666) age=1 idle=1 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22328 events=r cmd=subscribe user=default redir=-1 resp=2 id=218213 addr=10.0.14.45:48894 laddr=10.0.14.13:6379 fd=14 name=boydn-api-solr(SE.Redis-v2.6.86.49666) age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=ping user=default redir=-1 resp=2 r=10.0.14.10:6379 laddr=10.0.14.13:46154 fd=9 name= age=93266 idle=0 flags=M db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=62 qbuf-free=40892 argv-mem=27 multi-mem=0 rbs=1024 rbp=41 obl=0 oll=0 omem=0 tot-mem=42819 events=r cmd=set user=(superuser) redir=-1 resp=2 id=218214 addr=10.0.14.45:48910 laddr=10.0.14.13:6379 fd=15 name=boydn-api-solr(SE.Redis-v2.6.86.49666) age=1 idle=1 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22328 events=r cmd=subscribe user=default redir=-1 resp=2 id=218215 addr=10.0.14.7:51498 laddr=10.0.14.13:6379 fd=16 name=sentinel-6866bc27-cmd age=1 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=ping user=default redir=-1 resp=2 id=218216 addr=10.0.14.7:51514 laddr=10.0.14.13:6379 fd=17 name=sentinel-6866bc27-pubsub age=1 idle=1 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=138 obl=0 oll=0 omem=0 tot-mem=22328 events=r cmd=subscribe user=default redir=-1 resp=2 id=218217 addr=10.0.14.9:36654 laddr=10.0.14.13:6379 fd=18 name=sentinel-dc4d989a-pubsub age=1 idle=1 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=138 obl=0 oll=0 omem=0 tot-mem=22328 events=r cmd=subscribe user=default redir=-1 resp=2 id=218218 addr=10.0.14.3:41044 laddr=10.0.14.13:6379 fd=20 name=sentinel-2c433ea4-cmd age=1 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=ping user=default redir=-1 resp=2 id=218219 addr=10.0.14.3:41048 laddr=10.0.14.13:6379 fd=21 name=sentinel-2c433ea4-pubsub age=1 idle=1 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=138 obl=0 oll=0 omem=0 tot-mem=22328 events=r cmd=subscribe user=default redir=-1 resp=2 id=218206 addr=10.0.14.9:36628 laddr=10.0.14.13:6379 fd=19 name=sentinel-dc4d989a-cmd age=6 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=343 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=ping user=default redir=-1 resp=2 id=218209 addr=10.0.14.47:51224 laddr=10.0.14.13:6379 fd=10 name=boydn-api-frontend(SE.Redis-v2.6.86.49666) age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=info user=default redir=-1 resp=2 id=218230 addr=10.0.14.47:51330 laddr=10.0.14.13:6379 fd=22 name=boydn-api-frontend(SE.Redis-v2.6.86.49666) age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37632 events=r cmd=info user=default redir=-1 resp=2 id=218231 addr=10.0.14.47:51332 laddr=10.0.14.13:6379 fd=23 name=boydn-api-frontend(SE.Redis-v2.6.86.49666) age=0 idle=0 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37688 events=r cmd=subscribe user=default redir=-1 resp=2 ------ CURRENT CLIENT INFO ------ r=10.0.14.10:6379 laddr=10.0.14.13:46154 fd=9 name= age=93266 idle=0 flags=M db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=62 qbuf-free=40892 argv-mem=27 multi-mem=0 rbs=1024 rbp=41 obl=0 oll=0 omem=0 tot-mem=42819 events=r cmd=set user=(superuser) redir=-1 resp=2 argv[0]: '"SET"' argv[1]: '"UserId"' argv[2]: '"4"' argv[3]: '"PXAT"' argv[4]: '"1671029791661"' 1:S 14 Dec 2022 14:56:01.673 # key 'UserId' found in DB containing the following object: 1:S 14 Dec 2022 14:56:01.673 # Object type: 0 1:S 14 Dec 2022 14:56:01.673 # Object encoding: 1 1:S 14 Dec 2022 14:56:01.673 # Object refcount: 2147483647 ------ MODULES INFO OUTPUT ------ ------ CONFIG DEBUG OUTPUT ------ sanitize-dump-payload no list-compress-depth 0 io-threads 1 repl-diskless-load disabled proto-max-bulk-len 512mb lazyfree-lazy-server-del no replica-read-only yes activedefrag no lazyfree-lazy-user-flush no io-threads-do-reads no client-query-buffer-limit 1gb lazyfree-lazy-user-del no lazyfree-lazy-eviction no lazyfree-lazy-expire no slave-read-only yes repl-diskless-sync yes ------ FAST MEMORY TEST ------ 1:S 14 Dec 2022 14:56:01.674 # Bio thread for job type #0 terminated 1:S 14 Dec 2022 14:56:01.675 # Bio thread for job type #1 terminated 1:S 14 Dec 2022 14:56:01.675 # Bio thread for job type #2 terminated *** Preparing to test memory region 55f083265000 (2301952 bytes) *** Preparing to test memory region 55f08511d000 (344064 bytes) *** Preparing to test memory region 7fc398000000 (135168 bytes) *** Preparing to test memory region 7fc39c07f000 (22544384 bytes) *** Preparing to test memory region 7fc39d600000 (8388608 bytes) *** Preparing to test memory region 7fc39de00000 (2097152 bytes) *** Preparing to test memory region 7fc39e17f000 (2621440 bytes) *** Preparing to test memory region 7fc39e400000 (8388608 bytes) *** Preparing to test memory region 7fc39ec00000 (2097152 bytes) *** Preparing to test memory region 7fc39effd000 (8388608 bytes) *** Preparing to test memory region 7fc39f7fe000 (8388608 bytes) *** Preparing to test memory region 7fc39ffff000 (8388608 bytes) *** Preparing to test memory region 7fc3a0800000 (8388608 bytes) *** Preparing to test memory region 7fc3a1000000 (8388608 bytes) *** Preparing to test memory region 7fc3a188b000 (8192 bytes) *** Preparing to test memory region 7fc3a18a3000 (49152 bytes) *** Preparing to test memory region 7fc3a1a80000 (16384 bytes) *** Preparing to test memory region 7fc3a1aa2000 (16384 bytes) *** Preparing to test memory region 7fc3a1d96000 (16384 bytes) *** Preparing to test memory region 7fc3a1f77000 (8192 bytes) *** Preparing to test memory region 7fc3a1fa7000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible. === REDIS BUG REPORT END. Make sure to include from START to END. === Please report the crash by opening an issue on github: http://github.com/redis/redis/issues If a Redis module was involved, please open in the module's repo instead. Suspect RAM error? Use redis-server --test-memory to verify it. Some other issues could be detected by redis-server --check-system
