0

Producer.config

key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
max.in.flight.requests.per.connection=1
retries=3
acks=-1

Steps:

  1. started Kafka mirror on server1 and pushed 100 messages
  2. Verified the messages on server2.
  3. Closes kafka mirror on server1 and pushed 100 messages
  4. After few minutes starts the kafka mirror on server1.
  5. lot of messages which didnt get replicated to server2

I would appreciate if anyone could help me with the configuration details

Peyman Mohamadpour
  • 17,954
  • 24
  • 89
  • 100
Amita Rawat
  • 153
  • 1
  • 2
  • 6

1 Answers1

0

You need to ensure that the consumer.config is using the same group.id after restarting and that auto.offset.reset=earliest if you want to get all the messages after restarting

Sidenote: MirrorMaker (1.0) isn't meant for disaster recovery, only blindly copying of topics. MirrorMaker 2 has much more options for fault-tolerance between multiple clusters.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks cricket. Is there a way by which we can use timestamp as a parameter to decide the messages which needs to be replicated. I am coming from the fact that when my server1 goes down, entire traffic would shift to server2 and once server1 is up will have to mirror data from server2 to server1 and so on keep switching. With auto.offset.reset=earliest i will all the time get all messages irrespective of whether the messages are because of mirroring or actual message received from micro services. – Amita Rawat Nov 26 '19 at 12:11
  • No, MirrorMaker 1 doesn't offer this. MirrorMaker 2 can because it translates consumer offsets topic with associated timestamp information. The offset reset option only applies when a consumer group doesn't exist, but if it's always different you're always only reading the end of the topic – OneCricketeer Nov 26 '19 at 14:54