Redis encoding of objects and the size impact

Question

When I write some objects to redis I can get different memory usage stats. I want to understand how this happens.

Simple example:

127.0.0.1:6379> set a 1
OK
127.0.0.1:6379> MEMORY usage a
(integer) 49
127.0.0.1:6379> set a "1"
OK
127.0.0.1:6379> MEMORY usage a
(integer) 49
127.0.0.1:6379> set a \x01 <<<< Message packed of number 1
OK
127.0.0.1:6379> MEMORY usage a
(integer) 55
127.0.0.1:6379> set a '\x01' <<<< Message pack of number 1 but added '' 
OK
127.0.0.1:6379> MEMORY usage a
(integer) 55
127.0.0.1:6379> set a "\x01" <<<<< Message pack of number 1 but added "" 
OK
127.0.0.1:6379> MEMORY usage a
(integer) 52

and the encoding each time (as a embstr):

127.0.0.1:6379> set a 1
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f9858217ba0 refcount:2147483647 encoding:int serializedlength:2 lru:11691925 lru_seconds_idle:222
127.0.0.1:6379> set a "1"
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f9858217ba0 refcount:2147483647 encoding:int serializedlength:2 lru:11692152 lru_seconds_idle:1
127.0.0.1:6379> set a \x01
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f98563a1398 refcount:1 encoding:embstr serializedlength:5 lru:11692162 lru_seconds_idle:0
127.0.0.1:6379> set a '\x01'
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f98563a13e0 refcount:1 encoding:embstr serializedlength:5 lru:11692168 lru_seconds_idle:1
127.0.0.1:6379> set a "\x01"
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f98563a1788 refcount:1 encoding:embstr serializedlength:2 lru:11692177 lru_seconds_idle:1

A bit more complex value (string with a number)

127.0.0.1:6379> set a "abc123"
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f985821b260 refcount:1 encoding:embstr serializedlength:7 lru:11692243 lru_seconds_idle:3
127.0.0.1:6379> MEMORY usage a
(integer) 57
127.0.0.1:6379> set a \xa6abc123
OK
127.0.0.1:6379> MEMORY usage a
(integer) 61
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f985821b320 refcount:1 encoding:embstr serializedlength:11 lru:11692273 lru_seconds_idle:9
127.0.0.1:6379> set a "\xa6abc123"
OK
127.0.0.1:6379> MEMORY usage a
(integer) 58
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f985821b260 refcount:1 encoding:embstr serializedlength:8 lru:11692291 lru_seconds_idle:11
127.0.0.1:6379>

And a large JSON string (and then message packed) , the encoding is "raw"

127.0.0.1:6379> set a '[{"id":1,"first_name":"Kyrstin","last_name":"Ifill","email":"kifill0@livejournal.com","count":93,"ip_address":"182.218.153.253"}]'
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f98582b96f0 refcount:1 encoding:raw serializedlength:128 lru:11692490 lru_seconds_idle:2
127.0.0.1:6379> MEMORY usage a
(integer) 182
127.0.0.1:6379> set a '\x91\x86\xa5count]\xaafirst_name\xa7Kyrstin\xa9last_name\xa5Ifill\xa5email\xb7kifill0@livejournal.com\xaaip_address\xaf182.218.153.253\xa2id\x01'
OK
127.0.0.1:6379> MEMORY usage a
(integer) 197
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f98582b9700 refcount:1 encoding:raw serializedlength:143 lru:11692517 lru_seconds_idle:6
127.0.0.1:6379> set a "\x91\x86\xa5count]\xaafirst_name\xa7Kyrstin\xa9last_name\xa5Ifill\xa5email\xb7kifill0@livejournal.com\xaaip_address\xaf182.218.153.253\xa2id\x01"
OK
127.0.0.1:6379> DEBUG OBJECT a
Value at:0x7f98582b96f0 refcount:1 encoding:raw serializedlength:107 lru:11692535 lru_seconds_idle:2
127.0.0.1:6379> MEMORY usage a
(integer) 158

for_stack · Answer 1 · 2022-10-14T03:09:15.770

This answer based on Redis 4.0, on a bit-64 machine

In order to be memory efficient, Redis encodes value string in three methods:

Int: If the value string can be converted to an integer, e.g. -2^63 ~ 2^63, Redis saves the value as an integer. This is the most efficient encoding.
Embeded String: If the size of the value string is less than or equal to 44 bytes, Redis saves the string in the same chunk of the Redis object itself. This is more memory efficient than the Raw String encoding. Also, it's more cache-friendly. Check this for the reason.
Raw String: Otherwise, Redis uses the raw encoding.

In your cases:

set a 1 and set a "1": The value is a string that can be converted to 1. So Redis use Int encoding, i.e. encoding:int.
set a \x01 and set a '\x01': The value is a string with 4 characters （NOTE: since the string is enclosed in single quotes, it won't be escaped）. It cannot be converted to an integer, and its size is less than 44. So Redis use Embeded String encoding, i.e. encoding:embstr.
set a "\x01": Since the string is enclosed in double quotes, the value is escaped as a binary string, whose length, i.e. 1 bytes, is less than 44, and cannot be converted to integer. So Redis use Embeded String encoding, i.e. encoding:embstr.
Set a large json string: The value is a string whose size is greater than 44. So Redis use Raw String encoding, i.e. encoding:raw.

I think in "3" you mean it can be converted to an integer right? And what about the case where you mix a string and a binary string together? i.e. "mylongkeykeyabc1234" has a different memory footprint than "mylongkeykeyabc\xcd\x04\xd2" — Avba, Mar 25 '18 at 09:15
No, "\x01" CANNOT be converted to an integer. It's different from "1", whose ASCII code is "\x31". So `set a "\x31"` is the same as `set a 1`. — for_stack, Mar 26 '18 at 01:41
If you mix ascii string and binary string, Redis will use *Embeded String* or *Raw String* encoding, depends on the string length. In your case, `mylongkeykeyabc1234` is one byte longer than `mylongkeykeyabc\xcd\x04\xd2`, because of string escape. So they have different memory footprint. — for_stack, Mar 26 '18 at 01:48

Redis encoding of objects and the size impact

1 Answers1