0

I've currently got 22k sorted sets in Redis. Each key is in the following format:

monitor:<type>:feed:<MongoId>

Those sorted sets have a total of 11895225 items. The sets have a unix timestamp as the score and a MongoId as a member. 

<timestamp> - <MongoId>

The setup is extremly fast, but the database size is 1,8g. Is there any optimization I could do to reduce memory usage (keeping in mind that I need my data sorted by time)?

Peeter
  • 9,282
  • 5
  • 36
  • 53

2 Answers2

3

You could do some optimization, but don't expect too much of it. Each member has an overhead of about 64 bytes on top of the data itself (member-key + the score/float).

So the minimum memory size (without data) would already be around 0.77GB.

See this good answer by Didier Spazia: Redis 10x more memory usage than data

To do your optimization, you can use client-side hashing and/or client-side compression.

About your data design:

The sets have a unix timestamp as the key and a MongoId as a value.

Not quite sure what you mean here. The key is the correct term for the entire sorted set. What is your member score, and what is your member string? Do you rely on sorting by score? And, also on the lexicographical sorting by string? If both: not much you can do, except for maybe shortening the strings / tokens within the string. You can combine this with serializing your data using the MsgPack format. This is useful if you have numbers inside the string, which use far less bytes when serialized as MsgPack. If you use arrays (and not a serialized dict), you can again safe some space. With ints/floats, make sure your endianness matches the desired sorting though, when you use MsgPack in the member string and you need the lexicographical sorting characteristics of sorted sets.

Hope this helps, TW

Community
  • 1
  • 1
Tw Bert
  • 3,659
  • 20
  • 28
  • Sorry yeah, the unix timestamp is the score and mongoid is the member of the key. I'm not really following what you mean by "while optimizing". However I need the list sorted by score, I do not need lexicographical sorting of strings for the members. – Peeter Apr 09 '14 at 15:11
  • In that case, you can compress the member string data client-side. For example by using he `LZ4` algorithm. – Tw Bert Apr 09 '14 at 15:18
  • "while optimizing" : that was unclear indeed. I edited it. I also clarified the use of MsgPack in this scenario a litte. – Tw Bert Apr 10 '14 at 09:42
  • "Each member has an overhead of about 64 bytes on top of the data itself" This is no longer correct. Today in 2022 and redis 7 the overhead for the KEY is around 65 bytes but the memory usage for a member is close to the string len + score. – Grain Oct 31 '22 at 20:32
0

In addition to the excellent info from Tw, and by proxy Didier, another option which would involve not changing code would be to use a 32bit build of Redis. Of course, if you anticipate data growing beyond 2GB even under a 32b regime this wouldn't necessarily be a great option. However, for straight memory size reduction on "smaller" data sets it can be a simple solution.

At worst you could try it out with a copy of the data and run tests on it to see where your boundaries lie and how much memory you reduce but a decent rule of thumb is about half.

The Real Bill
  • 14,884
  • 8
  • 37
  • 39