0

Is there a concept like kafka routing key in redis streams? We have a number of microservice instances (let say 10) and we intend to create a consumer group, each consumer is one instance. We want messages of instance n will be read/retrieved only when instance n read/subscribe to the stream (based on some criteria in inside the message itself). is it possible?

SoT
  • 898
  • 1
  • 15
  • 36

1 Answers1

1

There is no way to reserve or route messages to a specific consumer in Redis: once a XREADGROUP command is issued, the returned messages are bound to the specified consumer - which must be specified upfront.

With that being said, Redis allows to change the ownership of a message through the XCLAIM command, once it is read. Thanks to that, you could somewhat mimic reservations / routings by having a specialized dispatcher consumer which continuously reads undelivered messages only (by way of the XREADGROUP command along with the special ID >) and assigns them through the XCLAIM command to the intended target consumer, based on their content. Each regular consumer reads messages delivered to it but still not acknowledged by way of the XREADGROUP command along with the starting ID 0, processes each message according to the business logic and finally XACK it to remove it from the PEL.

From the referenced documentation:

The special > ID, which means that the consumer want to receive only messages that were never delivered to any other consumer. It just means, give me new messages.

The target consumers, on the other side, would only read messages assigned to them (and not the undelivered ones) with the XREADGROUP command.

Efran Cobisi
  • 6,138
  • 22
  • 22
  • It means I will use XREAD to read undelivered messages, then use XCLAIM to assign the target and re-put it to redis, then the target will pull, right? – SoT Jan 01 '23 at 05:21
  • I would always use `XREADGROUP` / `XACK` to to that, so that Redis itself would keep track of the dispatched messages. I have just updated the answer to include a reference to the commands along with their documentation. – Efran Cobisi Jan 02 '23 at 06:31
  • I still do not quite understand completely. I assume that we will have a consumer group for my 10 instances (let say topic is '10_instances'), then we will have another consumer group for dispatcher (let say topic is 'response_from_somewhere'). The dispatcher will XREADGROUP / XACK, then XCLAIM to assign messages to '10_instances', right? – SoT Jan 03 '23 at 04:42
  • Not exactly: you need a single consumer group, which is shared by both your dispatcher **and** your 10 regular consumers. The dispatcher reads undelivered messages in a loop using `XREADGROUP ... >`, then `XCLAIM ...` to assign the message to the target regular consumer. Regular consumers read delivered but not acknowledged messages in a loop using `XREADGROUP ... 0`, then `XACK ...` each processed message. I have just updated my reply to make this clearer. – Efran Cobisi Jan 03 '23 at 07:41
  • Thanks, now I understand. sorry for my limited knowledge – SoT Jan 03 '23 at 07:55