1

I am consuming a redis list like this in python:

listitem = r.rpoplpush('mylist','mylist')

Strangely, the list gets empty randomly - for instance it will work without getting emptied for a month and then one fine day get emptied? What am I missing here? There is no other statement in my script which touches the script any way.

iLearn2016
  • 85
  • 9

1 Answers1

2

There is no such thing as an empty list in Redis, if a list is RPOP'ed all the way, the key is deleted.

So, one of these should be happening:

  1. The list mylist is getting emptied (LPOP, RPOP, LREM, LTRIM, etc) all the way until empty.
  2. The key mylist is being deleted (DEL, UNLINK, etc)
  3. The key mylist is being expired (EXPIRE, EXPIREAT, etc)
  4. The key is being evicted
  5. Data loss is occurring

If no redis-client is touching the key (1-3), then it must be 4 or 5.

See if you have some eviction policy set in your server with CONFIG GET maxmemory-policy.

Data loss may be occurring if you have no persistence and your server is restarted. Or if you are using more than a single instance (cluster or sentinel) and something is wrong. You can use the INFO command to see:

  • Server section: redis_mode and uptime_in_days.
  • Persistance section: relevant if uptime suggests the server restarted by the time you lost a key.
  • Memory & Stats sections: maxmemory_policy and evicted_keys will tell us if a policy is being applied
  • Replication & Cluster sections: if we still have no clue, start investigating this venue.
LeoMurillo
  • 6,048
  • 1
  • 19
  • 34
  • The only logical explanation could be this from Redis documentation for `AOF` In the past we experienced rare bugs in specific commands (for instance there was one involving blocking commands like BRPOPLPUSH) causing the AOF produced to not reproduce exactly the same dataset on reloading. These bugs are rare and we have tests in the test suite creating random complex datasets automatically and reloading them to check everything is fine. – iLearn2016 Jan 12 '20 at 11:46
  • @iLearn2016 That bug is hard to reproduce and there were no complaints about it in recent years. Have you loaded from AOF (i.e. restarted) and if so what version of Redis are you using? – Itamar Haber Jan 12 '20 at 12:20
  • @ItamarHaber No the server was not restarted. Here are info stats : https://pastebin.com/94adicC4 – iLearn2016 Jan 12 '20 at 15:41
  • It's a standalone with uptime of 509 days, noeviction, so it must be 1-3 – LeoMurillo Jan 12 '20 at 15:54
  • No it is not 1-3, I can tell you that with 200% surety, as I also stated it works for months without issue and then suddenly gets emptied. The script is same, there are 250 entries inside the list and the the single python script accessing the list is executed ~ 4000 times in a day. So if it is something in the script it would take less than a day to get empty. – iLearn2016 Jan 12 '20 at 16:01
  • Ok, your INFO pastebin shows connected_clients:32. If all use the same script then it is a tough one. If you think it may be a bug consider upgrading. 3.2.5 is from Oct-16. There is 3.2.13 from March-19 if you don't want to jump to 4 or 5. – LeoMurillo Jan 12 '20 at 16:24
  • @ItamarHaber What was the last version affected by the bug? My package manager allowed me to update to v 3.2.11 only. Also, can I upgrade to Redis v 5 without risking my data ?? – iLearn2016 Jan 14 '20 at 15:43
  • @iLearn2016 that bug affected AOF which requires a restart to manifest, which isn't the case iiuc. I don't remember when was the last time we saw it. Upgrading to 5 should be as easy as loading from the RDB, but you can/should test that before changing in production. – Itamar Haber Jan 15 '20 at 14:24
  • I searched internet and can't find an upgrade path/guide from redis 3 to 5. Any documentation to get started with? – iLearn2016 Jan 18 '20 at 05:27
  • See https://stackoverflow.com/questions/57876292/best-practice-to-upgrade-redis-with-sentinels – LeoMurillo Jan 18 '20 at 06:38