4

Bussiness Objective

I'm creating a dashboard that will depend on some time-series and I'll use Redis to implement it. I'm new to using Redis and I'm trying to use Redis-Streams to count the elements in a stream.

XADD conversation:9:chat_messages * id 2583 user_type Bot
XADD conversation:9:chat_messages * id 732016 user_type User
XADD conversation:9:chat_messages * id 732017 user_type Staff
XRANGE conversation:9:chat_messages - +

I'm aware that I can get the total count of the elements using the XLEN command like this:

XLEN conversation:9:chat_messages

but I want to also know the elements in a period, for example:

XLEN conversation:9:chat_messages 1579551316273 1579551321872

I know I can use LUA to count those elements but I want some REALLY fast way to achieve this and I know that using Redis markup will be the fastest way.

Is there any way to achieve this with a straight forward Redis command? Or do I have to write a Lua script to do this?

Additional information

I'm limited by AWS' ElastiCache to use the only Redis 5.0.6, I cannot install other modules such as the RedisTimeSeries module. I'd like to use that module but it's not possible at the moment.

Guy Korland
  • 9,139
  • 14
  • 59
  • 106
Alvaro Alday
  • 343
  • 3
  • 19

2 Answers2

3

While the Redis Stream data structure doesn't support this, you can use a Sorted Set alongside it for keeping track of message ranges.

Basically, for each message ID you get from XADD - e.g. "1579551316273-0" - you need to do a ZADD conversation:9:ids 0 1579551316273-0. Then, you can use ZLEXCOUNT to get the "length" of a range.

Itamar Haber
  • 47,336
  • 7
  • 91
  • 117
  • 1
    In this case wouldn't it be the same to use a Hash instead of a Redis-stream then? – Alvaro Alday Jan 21 '20 at 17:45
  • A hash wouldn't give you a timestamp-based id, and one may assume you were using other Stream features. Be careful with lex counting in the sense "1579551316273-10" appears before "1579551316273-2". Most likely not a problem for a char app as you need more than 10 entries in the same millisecond, but other applications may – LeoMurillo Jan 21 '20 at 18:20
  • That's very right @LeoMurillo - padding the sub-milli part with 0s is indeed needed in that case. – Itamar Haber Jan 23 '20 at 18:05
1

Sorry, there is no commands-way to achieve this.

Your best option with Redis Streams would be to use a Lua script. You will get O(N) with N being the number of elements being counted, instead of O(log N) if a command existed.

local T = redis.call('XRANGE', KEYS[1], ARGV[1], ARGV[2])
local count = 0
for _ in pairs(T) do count = count + 1 end
return count

Note the difference between O(N) and O(log(N)) is significant for a large N, but for a chat application, if tracked by conversation, this won't make that big of a difference if chats have hundreds or even thousands of entries, once you account total command time including Round Trip Time which takes most of the time. The Lua script above removes network-payload and client-processing time.

You can switch to sorted sets if you really want O(log N) and you don't need consumer groups and other stream features. See How to store in Redis sorted set with server-side timestamp as score? if you want to use Redis server timestamp atomically.

Then you can use ZCOUNT which is O(log(N)).

If you do need Stream features, then you would need to keep the sorted set as a secondary index.

LeoMurillo
  • 6,048
  • 1
  • 19
  • 34