Move data from Redis stream to a storage which is not in memory

Question

The https://redis.io/topics/streams-intro#capped-streams documentation mentions the capped streams to prevent memory overload:

...Sometimes it is useful to have at maximum a given number of items inside a stream, other times once a given size is reached, it is useful to move data from Redis to a storage which is not in memory...

However it only explains the redis capabilities on trimming the stream. I was not able to find any concept or a proven way to actually move the data from redis. I understand I can create a consumer to move all events to the unlimited place but the statement quoted above suggests that I should be able to move only old events in an efficient manner. Could you please share an idea of a solution?

Answer 1

IIUC you're looking to delete consumed messages, use case could be a replay or store it as historical data.

Redis as such does not offer a clean way to move data out of any Redis collection, capped stream just means you can trim the stream since it could potentially lead to out of memory.

The easiest way would be to add a consumer in the archive group that would consume from the stream and writes this data somewhere else. The consumer has to work for all required Redis streams where archival is required, this way you will always have data in secondary storage.

Now you would need some trim policy that would trim the collection, the easiest way could be trim it periodically like daily see my other answer How to define TTL for redis streams?