5

Using redis, there are a number of commands to retrieve entire data structures (LRANGE for lists, SMEMBERS for sets, ZRANGE for sorted sets, and HGETALL for hashes).

Only the hash has a method (HMSET) to insert multiple items with a single command.

All of the examples I've seen have shown just adding a single item at a time to a list (through RPUSH or LPUSH) or a set (through SADD/ZADD).

The more specific problem that I want to solve is to create lists and sorted sets containing database ids, these lists are unique to each user and contain a few hundred to a few thousand ids.

They are normally gathered from a database query, massaged a little bit in memory and then stored in redis for either paginating through (lists) or doing set based operations on to retrieve subsets (sets and sorted sets).

Currently, I'm iterating through the list and calling the appropriate add method for each element. This has the drawbacks of making multiple requests over the wire and repeating the key every time.

redis> RPUSH employee:ids 1000
(integer) 1
redis> RPUSH employee:ids 1001
(integer) 2
redis> RPUSH employee:ids 1002
(integer) 3
redis> RPUSH employee:ids 1003
(integer) 4
redis> del employee:ids
(integer) 1

I'm thinking that using a transaction with MULTI and EXEC could help to turn it into a single request, but that wouldn't help with the repeated key. 

redis> MULTI
OK
redis> RPUSH employee:ids 1000
QUEUED
redis> RPUSH employee:ids 1001
QUEUED
redis> RPUSH employee:ids 1002
QUEUED
redis> RPUSH employee:ids 1003
QUEUED
redis> RPUSH employee:ids 1004
QUEUED
redis> EXEC
1. (integer) 1
2. (integer) 2
3. (integer) 3
4. (integer) 4
5. (integer) 5

Is there something that I'm missing that would let me add elements to lists/sets in a single command, or without repeating the key every time?

I'm also using the jedis client library if that matters (or if there's another library that I can use from the JVM that'd be better).

Ted Naleid
  • 26,511
  • 10
  • 70
  • 81
  • As a side note, you might consider sorted sets instead of lists for pagination - list access time increases linearly with the offset, while sorted set access time is always proportional to the log of the size of the set. Lists are best used for stacks & queues, where you are usually accessing only at either end. – Mark Tozzi Jun 11 '13 at 16:33

1 Answers1

4

Setting up a transaction won't help here. The purpose of transactions is to ensure the commands get executed at the same time so you never have half a list on the server. They still get sent one at a time.

Are the multiple commands really causing performance issues? 1000 items isn't actually that many, and as long as you aren't doing something like opening a new connection for each item there shouldn't be any latency issues.

Tom Clarkson
  • 16,074
  • 2
  • 43
  • 51
  • 1
    I admit that I'm sort of pre-worrying about performance, but I haven't done any significant load testing with the current app. I'm partially just wondering if I'm doing it "right" or if there's something that I'm missing when I have a list/set in hand that I want to get in redis. If iterating through and adding one at a time is the way it's done, I'm cool with it. I might fork/submit patches to jedis to add methods that'll take an existing list/set and spin through the list for me. – Ted Naleid Jan 12 '11 at 16:04
  • 1
    No, you're not missing anything. Bulk add is something I could see being added to redis in the future, but it's not a significant enough improvement to get priority - 19 bytes at local socket speeds just doesn't make much difference. Adding support to your client is a good idea. You also have the option of not waiting for the response before sending the next item - it makes error handling harder, but should give you the same performance as a single command if you have latency issues. – Tom Clarkson Jan 12 '11 at 23:38