How to nest a list into a structure in Redis to reduce top level?

Question

I want to maintain some meta data in redis.

meta_key = build_key()
meta_data = {
    "user": 12345,
    "tag": "D12321341234123",
    }
res = redis_sip.hmset(meta_key, meta_data)

It works as expected.

Now I want to keep a list in this meta_data structure and be able to add element to the list.

For example:

meta_data = {
    "user": 12345,
    "tag": "D12321341234123",
    "items": []
    }

But it throws an exception right away:

redis.exceptions.DataError: Invalid input of type: 'list'. Convert to a byte, string or number first.

I reckon I can create a new key and use zadd to maintain a list. However I would like to minimise the number of key. It is because I need to quickly invalidate the keys once the user signs out. Keeping the keys to bare minimum can help me to

1) quickly evict the keys

2) avoid error because there are less keys to keep a tab on

Is there any way I can keep a list in a redis value and easily expand the list?

[This](https://stackoverflow.com/questions/16094574/alternatives-to-nested-structures-in-redis) is a similar question. You can try [redis-protobuf](https://github.com/sewenew/redis-protobuf) or [RedisJSON](https://github.com/RedisJSON/RedisJSON). — for_stack, Jan 30 '20 at 08:09

score 1 · Answer 1 · answered Jan 30 '20 at 05:33

I reckon I can create a new key and use zadd to maintain a list

yes that's the way I would do.

however if you want to minimize the key, you have no other way but to stringify the object to text and use set; also parse the string once you get it back.

this would limit you the way to access to single property of the object but you probably already know that.

if you opt for the additional list for each metaKey, you can use pipeline when insert/get/delete the item to reduce no. of round trips.

score 1 · Accepted Answer · answered Jan 30 '20 at 15:59

In most cases, SADD or ZADD with pipelining commands will be better. Use a MULTI/EXEC transaction if there is risk another client may get the key in between, therefore getting an incomplete object.

Stringify the list in a hash-field may be justifiable in a few cases.

Regarding 'Quickly evict the keys', make sure to use UNLINK instead of DEL.

If you choose to stringify, here is how to atomically support insert and remove to a JSON-encoded array in a hash field using Lua and Lua CJSON library:

Insert:

local items = cjson.decode(redis.call('HGET', KEYS[1], 'items'))
table.insert(items, ARGV[1])
return redis.call('HSET', KEYS[1], 'items', cjson.encode(items))

Remove by value:

local items = cjson.decode(redis.call('HGET', KEYS[1], 'items'))
local pos = -1;
for i, v in ipairs(items) do
    if ARGV[1] == v then
        pos = i
        break
    end
end
if pos == -1 then
    return -1
else
    table.remove(items, pos)
    return redis.call('HSET', KEYS[1], 'items', cjson.encode(items))
end

Usage Example:

> HGETALL meta_key
1) "user"
2) "12345"
3) "tag"
4) "D12321341234123"
5) "items"
6) "{}"
> EVAL "local items = cjson.decode(redis.call('HGET', KEYS[1], 'items')) \n table.insert(items, ARGV[1]) \n return redis.call('HSET', KEYS[1], 'items', cjson.encode(items))" 1 meta_key value1
(integer) 0
> HGETALL meta_key
1) "user"
2) "12345"
3) "tag"
4) "D12321341234123"
5) "items"
6) "[\"value1\"]"
> EVAL "local items = cjson.decode(redis.call('HGET', KEYS[1], 'items')) \n table.insert(items, ARGV[1]) \n return redis.call('HSET', KEYS[1], 'items', cjson.encode(items))" 1 meta_key value2
(integer) 0
> HGETALL meta_key
1) "user"
2) "12345"
3) "tag"
4) "D12321341234123"
5) "items"
6) "[\"value1\",\"value2\"]"
> EVAL "local items = cjson.decode(redis.call('HGET', KEYS[1], 'items')) \n local pos = -1; \n for i, v in ipairs(items) do \n     if ARGV[1] == v then \n     pos = i \n     break \n end \n end \n if pos == -1 then \n     return -1 \n else \n     table.remove(items, pos) \n return redis.call('HSET', KEYS[1], 'items', cjson.encode(items)) \n end" 1 meta_key value1
(integer) 0
> HGETALL meta_key
1) "user"
2) "12345"
3) "tag"
4) "D12321341234123"
5) "items"
6) "[\"value2\"]"
> EVAL "local items = cjson.decode(redis.call('HGET', KEYS[1], 'items')) \n local pos = -1; \n for i, v in ipairs(items) do \n     if ARGV[1] == v then \n     pos = i \n     break \n end \n end \n if pos == -1 then \n     return -1 \n else \n     table.remove(items, pos) \n return redis.call('HSET', KEYS[1], 'items', cjson.encode(items)) \n end" 1 meta_key value3
(integer) -1
> HGETALL meta_key
1) "user"
2) "12345"
3) "tag"
4) "D12321341234123"
5) "items"
6) "[\"value2\"]"

How to nest a list into a structure in Redis to reduce top level?

2 Answers2

Linked