3

We are using redis as caching. There were cases when some of the keys were getting deleted and we couldn't find the reason. I put redis keyspace/keyevent for CONFIG SET notify-keyspace-events KEgxe on this notification I am putting logs. But neither I can find in my core-logs nor in this redis-logs for some keys. They have some data in RDBMS, so we are sure that those keys must have come into Redis. We are very much into lost state, on what basis should we debug this.

Redis info says it has ~100mb in usage, which looks ok to me. How to persist data and keys.

theGamblerRises
  • 686
  • 1
  • 11
  • 27

2 Answers2

1

2 possible reasons:

  1. the client failed to write to Redis
  2. some client deletes these missing keys

You can try to subscribe everything about key changes: CONFIG SET notify-keyspace-events AKE, and log these changes into logs to figure out the problem.

for_stack
  • 21,012
  • 4
  • 35
  • 48
  • For 1st point, at least for some keys, I am sure that they were in redis, as I have seen those. For 2nd point, I used `KEgxe` for logging, which I thought will be sufficient for del, expire ... as logging `AKE` will be all the events. Anyway, I will put `AKE` and watch out. Thanks. – theGamblerRises Nov 05 '16 at 10:48
  • hi @for_stack, I tried with `AKE` as well, but I couldn't find anything in logs. It's just getting worse. I have some doubts, machine or redis got restarted, but in that case I should get connection error. and if redis max limit on number of keys or size, so policy is lru. My question is, if redis 'clears out' those keys, will that also come into keyspace notifications – theGamblerRises Nov 05 '16 at 15:18
  • @theGamblerRises if redis expires or evicts, it DOES send the notification. Is it possible, that your application unexpectedly called `flushdb` or `flushall`? Since these commands affect too many keys, Redis does NOT send any notification. – for_stack Nov 06 '16 at 01:06
  • You were right. Our ports were open and some external ip put flushall command. I had to put monitor and log those values. Lesson learnt. Thanks – theGamblerRises Nov 08 '16 at 21:46
  • @theGamblerRises OK, Some one deletes those keys on purpose. Normally, you should limit the access to Redis. For instance, use the `--bind 127.0.0.1` to forbid any connection from other host. – for_stack Nov 09 '16 at 02:22
1

Yeah someone is trying to hack into your redis server. Best to authenticate it, or even better restrict it by IP More details here - https://www.reddit.com/r/redis/comments/bv6uys/redis_keys_randomly_getting_deleted/

Harkirat Singh
  • 115
  • 2
  • 3
  • 2
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Adam Marshall Sep 07 '21 at 22:31
  • 1
    Fair point @AdamMarshall I The essential part of the link is that a lot of bots keep running looking for open redis instances. I personally was surprised that mine got hacked as soon as I spun it up. It explains how 75% of the open redis servers are being attacked using this attack. Also if you see the following keys in your redis cluster - backup1, backup2, backup3, backup4 , its most probably a bot that's messing with your redis instance – Harkirat Singh Sep 08 '21 at 12:22