0

I am trying to implement a way to wait until the table kafka-connector has caught up with the latest offset.

I've already tried implementing a session gap window operator on that table (kafka-connector to a compact topic) to wait for inactivity on the table and then collect the result locally (in an attempt to close the operator/task) but it became a global operator and the rest of my topology doesn't get executed (because this window operator creates a disjoint topology)

taricjain
  • 1
  • 1
  • What is the use case that you're trying to achieve? What do you mean with "caught up", since I'm assuming there will be new events that have arrived after you've started your Flink application. A stream is unbounded, meaning that it never ends. – Martijn Visser Dec 21 '22 at 09:37
  • So, I'm using the kafka connector to stream the history of a table from a log-compacted topic (stores events from a debezium service that streams all changes made to a table). thats why i want to wait until the entire history of the table has been played, to make sure I'm not serving the rest of the topology with old data. I hope that made sense – taricjain Dec 27 '22 at 21:41

0 Answers0