I'm using Redis Streams (via node's ioredis) on a project, and an automated test has revealed a potential race condition that I'm trying to work through a resolution for.
The problem boiled down is this: I have an event loop which continually makes an XREAD request on multiple streams over a single connection:
XREAD BLOCK 0 STREAMS stream1 stream2 $ $
When it receives a message on either stream, it updates its internal reference of the last message received on that stream, so that the next time through the event loop, the command looks like this:
XREAD BLOCK 0 STREAMS stream1 stream2 1234567890-0 $
The race condition I'm running into is that if a producer continually pushes messages onto stream1, stream 2 will never have an opportunity to receive any messages because every time we block we will immediately return the new messages from stream1 before we have the opportunity to wait for any incoming messages on stream2. Therefore stream2's last message received will always be $, so we will have no opportunity to "catch up" in between XREAD commands.
The actual implementation is one step more complicated, in that the set of streams we're listening to is dynamic and can be added to or removed from at will, so theoretically this issue could happen any time we're trying to listen to a new stream in addition to our existing ones.
I have been thinking about two options, both of which avoid usage of $:
Rather than use
$to establish the initial last message id received, use the current timestamp. This has the disadvantage of introducing the potential for clock skew between the Redis cluster and the consumers, but is simple to implement.Rather than use
$, make an XREVRANGE command prior to listening on a new stream to fetch the last message id currently on that stream, and then use that message id to begin the XREAD loop. This has the disadvantage of adding an extra request and some extra bookkeeping whenever we're interested in listening on a new stream, but is most likely the best option to guarantee consistency.
The reason I'm most confused here is that the redis streams documentation clearly recommends using $ as the recommended way of establishing an initial stream, and also clearly specifies the instructions for using XREAD to listen to multiple streams at once over a single connection, but makes no reference to this issue. Is there a different way of approaching this problem or a different implementation that I'm missing here?
