0

According to faust-streaming it is recommended to use Rocks DB for the fast recovery of the faust tables. Say there is one machine running a consumer with the faust table and Rocks DB. Rocks DB manages in-memory state of the table backed by the changelog topic for the fault-tolerant purposes.

If this machine dies and I have to start a new one, I will still have to read through the changelog topic to recreate the in-memory state of the table? If so, then the benefit of using Rocks DB comes when the next consumer is recreated on the same machine, right? Otherwise I will still have to read through the entire changelog topic?

Plesiosaurus
  • 1
  • 1
  • 1
    Yes, the value of using Rocks DB is the fast recovery time. If you have multiple workers monitoring a topic the topics will be distributed. If one of them dies, then those partitions it monitored need to be picked up by the others. If you use memory the still running workers read the changelog topic to get up to date, if you use rocks it effectively jumps straight to the latest. For a changelog of possibly millions of messages, that's a massive time saving. – Fonty Aug 22 '23 at 02:00

0 Answers0