0

With XREAD, I can get count from a specific ID (or first or last). With XRANGE, I can get a range from key to key (or from first - or to last +). Same in reverse with XREVRANGE.

How do I get by position? E.g. "give me the last 10 records in order"? XREVRANGE can do it with XREVRANGE stream + - COUNT 10, although it will be backwards in order, and XRANGE will give me the first 10 with XRANGE stream - + COUNT 10, but how do I:

  • get the last X in order?
  • get an arbitrary X in the middle, e.g. the 10 from position 100-109 (inclusive), when the stream has 5000 records in it?
deitch
  • 14,019
  • 14
  • 68
  • 96
  • What type of use case required you to get the x number of elements from some random position? – sonus21 Sep 11 '20 at 14:09
  • That's a fair question @sonus21. It is an existing API that allows users to retrieve records from index x to index y. It has a memory backend (just an array of records) and a file one, want to put Redis in it as well. – deitch Sep 13 '20 at 08:55
  • Well, I would just use LIST in that case, as it provides everything I need except consumer, I can build a consumer on top of that. You can use LIST and STREAM combination if you need stream at all. – sonus21 Sep 13 '20 at 14:28
  • Oh that is interesting. Instead of a stream, just a list of items? I didn't think of it. It doesn't answer this question directly, but instead goes to the core of my issue. How would you combine? – deitch Sep 13 '20 at 17:18

2 Answers2

1

Redis Streams currently (v6.0) do not provide an API for accessing their records via position/offset. The Stream's main data structure (a radix tree) can't provide this type of functionality efficiently.

This is doable with an additional index, for example a Sorted Set of the ids. A similar discussion is at https://groups.google.com/g/redis-db/c/bT00XGMq4DM/m/lWDzFa3zBQAJ

The real question is, as @sonus21 had asked, what's the use case.

Itamar Haber
  • 47,336
  • 7
  • 91
  • 117
  • Well, this isn't the answer I want, but it is correct. I described the use case in a response comment to @sonus21 above. I don't really have full control over the API. – deitch Sep 13 '20 at 08:57
  • By the way, how about the last X but in order? I can use `XREVRANGE` to get the last 10 in reverse order, but how about in order? Oh, I guess I can use it and get the count of 1, then get the ID, then use `XRANGE`? – deitch Sep 13 '20 at 08:58
1

It is an existing API that allows users to retrieve records from index x to index y. It has a memory backend (just an array of records) and a file one, want to put Redis in it as well.

We can just use Redis List to get the items randomly and at any offset using LRANGE

Attaching consumers to Redis LIST is easy but it does not come with an ack mechanism like a stream, you can build an ack mechanism in your code but that could be tricky. There could be many ways to build an ack mechanism in Redis, see one of them here Rqueue.

Ack mechanism is not always required but if you just want to consume from the LIST, you can use BLPOP/LPOP commands to consume elements from the list, that could be your simple consumer, but using BLPOP/LPOP commands would remove entries from the LIST so your offset would be dynamic, it will not always start from the 0. Instead of using BLPOP/LPOP, if you use LRANGE and track the offset in a simple key like my-consumer-offset, using this offset you can build your consumer that would always get the next element from the list based on the current offset.

Using LIST and STREAM to get random offset and streaming feature. Most important part would be you should use Consumer group so that streams are not trimmed. During push operation you should add element to STREAM as well as LIST. Once elements are inserted your consumer can work without any issue in a consumer group, for random offset you use LIST to get elements. Stream can grow as it'd be in append only mode, so you would need to periodically trim stream. See my other answer for deleting older entries

sonus21
  • 5,178
  • 2
  • 23
  • 48