2

I have a Flink application that was implemented following the event-sourcing paradigm. Both events and commands are stored in several Kafka topics.

The application has two startup modes: recovery and production. First, the recovery mode is used to recover the application state (a savepoint) from events topics. In this mode, commands are not read at all. Once all event topics have been processed, a savepoint is triggered manually (from command line) and the application is stopped. Then, the yarn process is started in production mode. In that mode the application processes both events and commands.

I would prefer to execute this process programatically. To do it, several questions arise... How can the application itself:

  1. detect that kafka sources have been fully readed?
  2. trigger a savepoint programmatically?
  3. stop and start programmatically?

Thank you!

user2108278
  • 391
  • 5
  • 17
  • Kafka consumer can throw an exception automatically when no more messages are available by setting consumer.timeout.ms to a value. Could it be used to 1? How to handle the exception gracefully? – user2108278 Nov 08 '16 at 11:56
  • Related link: http://stackoverflow.com/questions/29452569/why-isnt-this-kafka-consumer-shutting-down – user2108278 Nov 08 '16 at 12:06
  • For step 2, to manually create a savepoint and cancel a job: ./bin/flink cancel -s [targetDirectory] – user2108278 Nov 08 '16 at 12:38

0 Answers0