0

I am new to Redis and found that there are various data structures supported in Redis and based on requirement we can choose any one of them. My requirement is to insert strings one by one to a list(key-list) and wants to retrieve(and remove) all at once. And also I want to do this very frequently, so trying to find an optimal way. Which data structure/way will be better for this? Thank you in advance

P.S: I don't want to remove the key when retrieving, I just need to retrieve and empty the list.

RaR
  • 3,075
  • 3
  • 23
  • 48

3 Answers3

1

Sounds like you should use a List. Add to the list with either LPUSH or RPUSH, then retrieve everything with LRANGE and DEL the key.

P.S. A key in Redis, such as one storing the List, can't be empty. Once you remove all of the list's members, the key itself no longer exists hence you can just delete it instead of trying to empty it.

Updated with an answer to the OP's comment: not really, there are no free lunches and regardless the method you'll have to do O(N) reads and deletes. There are cases where performing just one iteration is preferable, e.g. to reduce network communication, but this isn't one of them.

Any way, the closest you can get in terms of functionality to a combination of both is with Lua. Note, however, that this will not necessarily perform better then LRANGE & DEL:

$ cat popall.lua 
local r={}
local e=redis.call('LPOP', KEYS[1])
while e do
    r[#r+1]=e
    e=redis.call('LPOP', KEYS[1])
end
return r
$ redis-cli LPUSH list 1 2 3 4 5 6 7 8 9
(integer) 9
$ redis-cli --eval popall.lua list
1) "9"
2) "8"
3) "7"
4) "6"
5) "5"
6) "4"
7) "3"
8) "2"
9) "1"
$ redis-cli EXISTS list
(integer) 0
Itamar Haber
  • 47,336
  • 7
  • 91
  • 117
  • `LRANGE` and `DEL` are both O(N) operations. `POP`ing until list gets empty will take N * O(1) operations. Is there a way to combine both? (ie) pop and return all at once, taking single O(N) – RaR Nov 28 '16 at 09:07
1

You likely want to use a simple list to achieve this. There is one reason you might want to use a set but I'll get to that later.

You will use LPUSH to add items to the list. Note that the list doesn't need to exist for you to be able to perform this operation for the first time. This is a O(1)(constant time) operation so that's as fast as you can get. To retrieve all items you will use LRANGE 0 -1. This will end up being O(N) which is the same as SMEMBERS (the analogous set operation). Finally you will use DEL which again will be O(N) and will have the same performance whether you use a set or list.

As I mentioned, there is one situation where you would use a set over a list. That is when you want to prevent duplicates but do not care about order. In this case you will add members with SADD and retrieve all members with SMEMBERS. Using a set has the exact same performance as using a list except you are trading order for uniqueness. In the case that you want to prevent duplicates but also care about order, you will want to use sorted sets. The operations will be a little bit more complex as you will need to keep track of the scores, but it's still fairly simple. You will also take a very minor performance hit by using sorted sets. This is not something to worry about as Redis is very fast.

Curtis Maddalozzo
  • 303
  • 2
  • 5
  • 1
    First of all, thank you for such a wonderful explanation. The way you suggested, using list or set, takes minimum of two O(N) operations. Is there no way to do it in a single O(N) operation. `(L/R)POP` removes one and returns the same, which is an O(1) operation. Like that, isn't there a way to pop all and return all while taking O(N). – RaR Nov 28 '16 at 09:03
  • 1
    @RaR No worries. That's a great point. It seems that you can achieve the best performance by repeatedly popping as you said, rather than doing a range then a delete. I don't think there is any sort of "bulk pop" for any of the Redis data structures. If you're list sizes are going to be really big (like in the millions) and this operation will happen frequently, it might be worth doing the manual pop method. Otherwise for simplicity sake it's probably best to just stick with the range/delete method. – Curtis Maddalozzo Nov 28 '16 at 16:00
1

You could also use APPEND if you have some character that you can use as separator. It's O(1). Then you can get the whole string and reset it using GETSET. The downside is that you will have to turn it into a list again in your own code if that's what you need.

Erik Dubbelboer
  • 127
  • 1
  • 6