redis collection atomic lpop all

Question

redis list, a producer keeps lpush. In another thread, consumers periodically take all out from the list, and categorize elements. Because the producer keeps pushing, so the taking-all-out must be done atomically. So is there an effective way to do this? spring-data-redis could be used.

// producer
getOpsForList.push(k, v);

// consumer
alist = range(k,0,-1); // take all out
alist.parallelStream() // during which a producer thread could push but I hope it is "blocked".
delete(k);  // list is now empty and push from producer is unblocked.

multi and exec does not achieve my goal, because it actually submits lrange, lpush and delete just in one transaction. So far, the only way I could think of, is keeping lpop and add returned into alist until list is empty.

EDIT, this is what I think: when you want to be sure an operation is ran only once, using watch:

watch key
val = get key
val = val + 1
multi
set key val
exec

when you want to be not "interrupted" (not multithreading interrupt), and don't care how many times it runs, transaction (multi and exec) is enough.

multi
val = lrange key 0 -1
delete key
exec

val is still a list after it finishes, like what is said in official-doc

All the commands in a transaction are serialized and executed sequentially. It can never happen that a request issued by another client is served in the middle of the execution of a Redis transaction.

Beyond redis, I took the data operation list.stream.parallelism out, and the function now only focuses on the data getter, which is exactly like the last code paragraph. ;)

holi-java · Accepted Answer · 2017-09-11T13:39:26.467

0

A good example to illustrate how WATCH can be used to create new atomic operations otherwise not supported by Redis is to implement ZPOP, that is a command that pops the element with the lower score from a sorted set in an atomic way.

There is a implementation for ZPOP in documentation as below:

WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC

What you need to do is repeat the operation above If EXEC fails (i.e. returns a Null reply). the producer operation lpush is atomic, so it needn't to use watch command. for example:

// consumer pesudo code
do {
  watch(k);
  transaction = multi();
  alist = transaction.range(k,0,-1); 
  transaction.delete(k);  
  status = get status of transaction.exec();
} while(status == null);

alist.parallelStream()

edited Sep 11 '17 at 13:39

answered Sep 11 '17 at 13:27

holi-java

29,655
7
72
83

pls, put the opening text in a comment (if at all). SO, is no place for flirting, casanova ;)! – Willi Mentzel Sep 11 '17 at 13:34
@holi-java watch would submit an execution when the watching target has not changed. While this is not exactly what I want. I wish when consumers accept all elements, producers's push operation could be blocked, and only recovered when consumers finish (i.e., after delete). But I will compare your solution with the lpop loop. – Tiina Sep 12 '17 at 00:19
@holi-java producer pop randomly but quite often. spring-data-redis has `pushAll` so I am looking for a way to achieve `popAll` – Tiina Sep 12 '17 at 01:05
@Tiina why does producer need to pop element out of list? – holi-java Sep 12 '17 at 01:16
@Tiina if your reqiurement is pop all element from consumer you can set 2 keys one is `offset` starts at 0, another is `n` obtained by `llen`, then you can use pipeline to pop elements in each chunk(e.g: chunk size is 100) until `offset >= n`. – holi-java Sep 12 '17 at 01:23
@Tiina hi, have you solved the problem? I'd like to know. – holi-java Sep 12 '17 at 12:14
@holi-java please see EDIT part in my po. – Tiina Sep 14 '17 at 01:43
@Tiina do you care about the order popped from the list? are there huge of elements popped from list? – holi-java Sep 14 '17 at 10:24

redis collection atomic lpop all

1 Answers1