3

I have 1000 to 10,000 keys stored in Redis, their value type is list. When a new item is added to any one of the existing lists, I need my golang program to be notified. Once notification is received I need to spawn a new goroutine and perform a small operation.

I am using redigo for redis connection pool.

What is the best approach to solve this problem, without overloading the Redis instance?

Vatine
  • 20,782
  • 4
  • 54
  • 70
Madhu Sudhan Reddy
  • 77
  • 5
  • 2
    There's no efficient way to do this with lists. I'd probably change the thing pushing to the list to both push to a list and to send a pub/sub message saying which list it just pushed to. Wrap that up in a lua function and it will be pretty darn efficent. More efficient than opening 10k sockets each blocking on BLPOP or whatever you are doing to read the lists – David Budworth Jan 24 '16 at 03:39

2 Answers2

5

You can enable Redis' keyspace notifications and subscribe to the relevant events on the keys/patterns that interest you.

More details can be found in the documentation: http://redis.io/topics/notifications

Itamar Haber
  • 47,336
  • 7
  • 91
  • 117
2

I haven't tried this, but as speculation, I would use Redis's Lua to implement 2 new commands - MSR_PUSH and MSR_POP - which would do the following respectively:

-- MSR_PUSH
redis.call("PUSH", KEYS[1], ARGV[1])
redis.call("PUBLISH", "notify", KEYS[1])

and:

-- MSR_POP
local v = redis.call("POP", KEYS[1])
if v then
  redis.call("PUBLISH", "notify", KEYS[1])
end
return v

So, these Lua scripts update the lists as you normally do, but then also publish the keyname that was updated to the notify pub/sub, which will then allow a watching script (golang) to do something. You could also just push to another queue, and long poll that.

This link has more information on Lua with Redis: https://www.redisgreen.net/blog/intro-to-lua-for-redis-programmers/

Itamar Haber
  • 47,336
  • 7
  • 91
  • 117
Daniel
  • 7,006
  • 7
  • 43
  • 49