Implementing Queue with unique elements in Redis

Question

I need to implement the Queue in Redis with unique elements. Currently, I am using the Redis List to implement Queue (LPUSH, RPOP) and Redis Sortedset/set to implement uniqueness.

`  
def push(key):
    if redis_cache.zadd('UNIQUE', key, 1):
        redis_cache.lpush('QUEUE', key)
    else:
       print "Key Exist"
`
`
def pop():
    key = redis_cache.rpop('QUEUE')
    redis_cache.zrem('UNIQUE', key)
    return key
`

With high load/request of keys the Redis Cache server using more CPU. and also the above approach taking more memory size (The same key is sored in both List and Sortedset) Is there any other way to implement the Redis Queue with key uniqueness?

score 4 · Answer 1 · answered Nov 15 '19 at 16:43

4

I'd actually look into using a single Sorted Set, setting the scores as timestamps and using ZADD's "NX" flag. This will provide both ordering and uniqueness.

answered Nov 15 '19 at 16:43

Itamar Haber

47,336
7
91
117

Good idea! Although the order is not very accurate (items added in the same second or millisecond are ordered by key), and client might cheat (by specifying a previous timestamp), in most case, it should be a good solution. – for_stack Nov 16 '19 at 06:55
This is the good idea. I think my problem will get solve with this solution. Thank you. – Rahul.Shikhare Nov 19 '19 at 07:00
There are two issues with using timestamps as the `score`: 1) time may go backwards ( due to NTP drift for example). 2) You may get the same timestamp if the `ZADD`s are close to each other - ordering then will be lexicographic which is probably not what you want – David Soroko May 06 '20 at 16:05

score 1 · Answer 2 · answered Nov 15 '19 at 08:29

First of all, you should use a Set, instead of Sorted Set to check if the key already exist. Sorted Set uses more memory than Set.

Secondly, you code is NOT atomic. If you want to make it atomic, you need to use Lua script or transaction.

Is there any other way to implement the Redis Queue with key uniqueness?

Random Order Queue

If you don't care the order of the queue, i.e. you don't need a FIFO, you can use a single Set without a List to implement a queue with random out order. This solution will save more memory, and much faster, since it only need to send one command to Redis.

// in queue
SADD UNIQUE key

// out queue, pop a random member.
SPOP UNIQUE

Yep, a race condition exists in the pop operation. – Itamar Haber Nov 15 '19 at 16:44 — Itamar Haber, Nov 15 '19 at 16:44

score 0 · Answer 3 · answered Nov 15 '19 at 07:58

According to this answer, you must choose between making multiple operations - Cost in time (remove then add again if found during removal) or maintaining a separate set - Cost in memory (your approach). LREM is O(N), where N is the length of the list, but SREM complexity for one element is O(1) since N is the number of members to be removed. And SISMEMBER is always O(1).

Implementing Queue with unique elements in Redis

3 Answers3

Random Order Queue