0

I am using Spring Data Redis. If a Redis SET has millions of values, will fetching its members (am using members() function) create a Java Set in the heap with a million values? Or is values fetched as and only required?

If all the values are fetched at one go, will it throw some out of memory error if the SET is huge? If so, how can I overcome that? I have got the same doubt with the range() of LIST and ZSET.

sinujohn
  • 2,506
  • 3
  • 21
  • 26

2 Answers2

1

SD Redis v 1.2 implementation for RedisSet has no lazy loading implementation.

But commands like add and remove are delegated to the underlying RedisConnection performing operations at the server without affecting any local data.

Using RedisSet.iterator() will execute SMEMBERS and load the entire response into memory, which is likely to use quite a lot of memory.

Same goes for the List and Maps implementation.

Christoph Strobl
  • 6,491
  • 25
  • 33
  • So I'll have to use LIST or ZSET instead of SET and then use range() to fetch limited number of members so as not to overwhelm my heap size? – sinujohn May 20 '14 at 05:35
  • just found out SCAN command. Don't know whether Spring Data Redis supports it currently – sinujohn May 20 '14 at 11:19
  • 1
    `SCAN` is currently not supported so you'd have to directly use the underlying redis connection (jedis directly supports scan, and you can emulate it for lettuce). But we're already working on SCAN support. See [Pull Request: #71](https://github.com/spring-projects/spring-data-redis/pull/71) – Christoph Strobl May 20 '14 at 12:22
0

The documentation doesn't mention anything about lazy loading, so it should be understood that all the elements of the set are fetched at once. Which will result in an OutOfMemoryError if you have huge sets.

Kayaman
  • 72,141
  • 5
  • 83
  • 121