0

Using Redis stream we can have pending items which aren't finished by some consumers. I can find such items using xpending command. Let we have two pending items:

1) 1) "1-0"
   2) "local-dev"
   3) (integer) 9599
   4) (integer) 1
2) 1) "2-0"
   2) "local-dev"
   3) (integer) 9599
   4) (integer) 1

The problem that by using xpending we can set filters based on id only. I have a couple of service nodes (A, B) which make zombie check: XPENDING mystream test_group - 5 1 Each of them receives "1-0" item and they make xclaim and only one of them (for example A) becomes the owner and starts processing this item. But B runs xpending again to get new items but it receives again "1-0" because it hasn't been processed yet (A is working) and it looks like all my queue is blocked.

Is there any solution how I can avoid it and process pending items concurrently?

Max Grigoriev
  • 1,021
  • 2
  • 13
  • 29

1 Answers1

0

You want to see the documentation, in particular Recovering from permanent failures.

The way this is normally used is:

  1. You allow the same consumer to consume its messages from PEL after recovering.
  2. You only XCLAIM from another consumer when a reasonably large time elapsed, that suggests the original consumer is in permanent failure.
  3. You use delivery count to detect poison pills or death letters. If a message has been retried many times, maybe it's better to report it to an admin for analysis.

So normally all you need is to see the oldest age in PEL from other consumers for the Permanent Failure Recovery logic, and you consume one by one.

LeoMurillo
  • 6,048
  • 1
  • 19
  • 34
  • `So normally all you need is to see the oldest age in PEL from other consumers for the Permanent Failure Recovery logic, and you consume one by one.` - that's why I'm talking about. If you have a couple instances and all of them are looking for such permanent failed messages then all of them will receive the same item until the item isn't processed. – Max Grigoriev Aug 27 '20 at 18:03
  • No, because you are looking at the oldest. As soon as another client reclaimed a message, the age for that one would restart from zero – LeoMurillo Aug 27 '20 at 19:43
  • The age is changed but not the id which is linked to timestamp. And when instance `A` makes `xclaim` the item `"1-0"` is still pending but owner is just changed to `A`. So `xpending` on `B` will receive the oldest item which is `"1-0"`. B will receive a new item only when A processes the `"1-0"` – Max Grigoriev Aug 27 '20 at 19:49