0

Say the kafka spout fetched some messages and at that time the spout task is restarted. Will the fetched messages be lost? I'm a starter on trident and my question is for the trident kafka transactional/opaque spout.

Thanks in advance!

Adrian Liu
  • 385
  • 3
  • 13

1 Answers1

1

I believe as long as the messages are not ack they wont be commited as consumed and the spout will replay them when started .

On the other hand if you configure your spout to read from the beginning then Kafka-sppout will fetch them from the starting offset point.

user2720864
  • 8,015
  • 5
  • 48
  • 60
  • Thanks for the reply. I didn't make it clear but my questions is what if the spout fetched the messages and then crashed without emitting the messages? In this situation storm's replay mechanism won't work. Then a new kafka spout is started by storm, what will be the offset to read from? Thanks! – Adrian Liu Feb 28 '15 at 02:54
  • If u set it to read from beginning it will start with the first offset. Otherwise if the spout crashed before committing the offset to zookeeper the message won't be considered as processed and the time u start the spout it will be replayed – user2720864 Feb 28 '15 at 15:10