4

We are setting up MirrorMaker 2.0 for Kafka. If I understand correctly, topic offsets are not equal in replicated cluster. This is not a problem for regular Kafka app as consumer groups get replicated too. Flink stores Kafka offsets internally in state - I assume after job restart with state, things can go awry. Is there a way to set up Flink so that we can recover from cluster failure on replicated Kafka cluster? I think we should migrate state somehow but have no experience doing that.

bottaio
  • 4,963
  • 3
  • 19
  • 43

2 Answers2

0

I don't think that migrating the state will help in this case. I can see two ways of tackling this issue, but none of them is perfect:

  1. Technically, you could disable storing offsets on checkpoint and enable enable.auto.commit in consumer, but I think this is a little dangerous in terms of possible losing data.
  2. You can extend the FlinkKafkaConsumer and modify the open function, so that it will ignore the offsets stored in state and instead it will use fetcher to fetch the offsets. This should work fine I think, since the offsets are commited to Kafka on checkpoint if the checkpointing is enabled.
Dominik Wosiński
  • 3,769
  • 1
  • 8
  • 22
0

MirrorMaker 2 manages to sync offsets via an internal topic and uses the timestamp of the messages to account for syncing and drift in the case of failover.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • ok, but Flink is not aware of that as it is not using Kafka for storing offsets at all to guarantee exactly-once semantics – bottaio Mar 25 '20 at 08:30
  • @bottaio You are comparing apples and oranges. You cannot setup cross-DC replication alone with Flink with a typicaly consume-produce action. At least not without offset mismatch, which needs to be addressed externally using timestamps and you can seek consumer groups back on times – OneCricketeer Mar 25 '20 at 11:45
  • I don't know guys if you have experience with Flink but when starting from savepoint / checkpoint offsets stored in Kafka are basically useless - Flink is going to use offsets stored internally. Savepoint / checkpoint might point to any point in time - and there is just a value stored for latest checkpoint in Kafka. Moreover, this value might not even correspond to the latest successful checkpoint - there are cases when Flink keeps on processing data and fails to create checkpoints. In such case offsets stored in Kafka could in rare cases include data from failed checkpoint. – bottaio Mar 25 '20 at 12:54
  • I am expecting and answer from somebody who have experienced such failure scenario or is aware of the possible set up. – bottaio Mar 25 '20 at 12:56
  • AFAIK, the state store is configurable. At least in Spark it is... You _can_ use the data stored in Kafka, it is just _not the default_. Personally, no, I have not used Flink, but your question seemed to be asking about MM2 anyway... – OneCricketeer Mar 25 '20 at 13:02
  • The heartbeat and internal topic descriptions are documented here https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-InternalTopics – OneCricketeer Mar 25 '20 at 13:04
  • 2
    Flink shouldn't generally store the offsets for failed checkpoints as the actual offset storing is done after the checkpoint signalizes it has finished. – Dominik Wosiński Mar 25 '20 at 19:10