Iterating through keys in Redis

Question

I have just started with Redis. My DB contains about 1 billion records. Using HKEYS * results in an out of memory error.

Is there a way to iterate through keys? Something like HKEYS * but with a limit n?

Edit:

I am now using a loop which matches a pattern

for c in '1234567890abcedf':
    r.keys(c + '*')

You meant `KEYS *`, right? `HKEYS` is for listing keys of a hash. Anyway, if you use `KEYS` in your code there is a big chance you're doing something wrong, i.e. not using Redis as it should be used. `KEYS` should be treated as a debugging function only. You're probably not using the right redis datatype. Please describe your problem (e.g. why you need to list the keys) and I might be able to help choose the right redis database that matches your problem. — Michał Kwiatkowski, Feb 02 '12 at 12:48
See the answer of http://stackoverflow.com/questions/9127736/redis-sorted-sets-and-best-way-to-store-uids/9195219#9195219 - you have an example of scalable key iteration there. — Didier Spezia, Feb 09 '12 at 15:11

score 8 · Answer 1 · edited Oct 14 '18 at 09:33

8

Available since Redis 2.8.0 are the cursor based Redis iteration commands (SCAN, HSCAN etc) that let you iterate efficiently over billions of keys. For your specific case, the start using HSCAN instead of HKEYS/HGETALL. It is efficient, cheap on server resources and scales very well. You can even add a pattern to HSCAN unlike HKEYS.

e.g.

127.0.0.1:6379> HMSET hash0 key0 value0 key1 value1 entry0 data0 entry1 data1
OK
127.0.0.1:6379> HSCAN hash0 0 MATCH key* 
1) "0"
2) 1) "key0"
   2) "value0"
   3) "key1"
   4) "value1"
127.0.0.1:6379> HSCAN hash0 0 
1) "0"
2) 1) "key0"
   2) "value0"
   3) "key1"
   4) "value1"
   5) "entry0"
   6) "data0"
   7) "entry1"
   8) "data1"

edited Oct 14 '18 at 09:33

Martijn Pieters

1,048,767
296
4,058
3,343

answered Nov 07 '16 at 06:14

Vaibhaw

598
8
17

upvoted! i understand how this works but my keys dont have a pattern, they are integers from 1 to 1000 and contain a large string each, i dont want to hgetall in memory ,is there any way i can loop through them – PirateApp Jun 30 '18 at 12:02
1

@PirateApp Don't think there is a way that the MATCH command will take a pattern that will match the sort of keys you have. Folks usually create a "namespace hierarchy" in Redis keyspace to bring some order to their store. Consider adding prefixes to your keys. On the other hand, if you already know your keys, you can fetch them by specifically too, no? – Vaibhaw Jul 04 '18 at 02:57

score 1 · Answer 2 · answered Apr 24 '14 at 19:34

1

You can't iterate over redis keys directly, but you can accomplish something very similar by transactionally writing the key portion of your key-value pair to a sorted set at the same time you write your key-value pair.

Downstream, you would "iterate" over your keys by reading n keys from the sorted set, and then transactionally removing them from the sorted set at the same time as you remove the associated key-value pair.

I wrote up an example with some C# code here: http://rianjs.net/2014/04/how-to-iterate-over-redis-keys/

You could do this in any language that has a redis library that supports transactions.

answered Apr 24 '14 at 19:34

rianjs

7,767
5
24
40

SCAN did not exist when this question was asked. – rianjs Dec 29 '16 at 17:56
2

I've added a point to this answer since it was correct at the time. People should be aware of temporal answers before marking them down. – Lloyd Moore Jan 05 '17 at 08:43

Lloyd Moore · Answer 3 · 2017-01-05T08:42:41.223

0

Sorry, at the current time, year 2012, the simple answer is no, however, with lua scripting you could do it, although that is not direct redis in the strictest sense.

edited Jan 05 '17 at 08:42

answered May 11 '12 at 07:46

Lloyd Moore

3,117
1
32
32

I believe this was not available in 2012 @Cosmin – Lloyd Moore Jan 05 '17 at 08:40

Niels · Answer 4 · 2015-03-21T23:20:34.073

0

For iterating through keys:

SCAN cursor [MATCH pattern] [COUNT count]

http://redis.io/commands/scan

For iterating through the values of a hash

HSCAN key cursor [MATCH pattern] [COUNT count]

http://redis.io/commands/hscan

edited Mar 21 '15 at 23:20

answered Mar 21 '15 at 14:00

Niels

1,513
3
20
29

Iterating through keys in Redis

4 Answers4