2

I have a problem with replication of topics from cluster A to cluster B with Mirrormaker 2. I want to replicate these topics from latest offset instead of replicating them from the beginning. But when I set auto.offset.reset=latest nothing happens and replication still happens from the beginning of topics instead of an end. Do I set it wrong?

This is my whole MM2 configuration:

clusters = A, B

A.bootstrap.servers = broker-01A:9092
B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092

replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy

#Enable replication between clusters and define topics which should be replicated
A->B.enabled = true
A->B.topics = .*
A->B.replication.factor=3
A->B.emit.heartbeats.enabled = true
A->B.emit.checkpoints.enabled = true

auto.offset.reset=latest
refresh.topics.enabled=true

heartbeats.topic.replication.factor=1
checkpoints.topic.replication.factor=1
offset-syncs.topic.replication.factor=1

config.storage.replication.factor = 1
offset.storage.replication.factor = 1
status.storage.replication.factor = 1

UPDATE

After changing the auto.offset.reset=latest to A.consumer.auto.offset.reset=latest the auto.offset.reset=latest configuration started to appear in Mirrormaker logs. But still no luck. Still mirroring is done from the start.

Scourrge
  • 83
  • 7
  • Try `consumer.auto.offset.reset=latest`. If you look at the Kafka connect documentation, it mentions you need the prefix – OneCricketeer Jun 13 '22 at 13:31
  • 1
    Yeah. After reading the docs I've tried `consumer.auto.offset.reset=latest` and it didn't help. After trying `clusterName.consumer.auto.offset.reset=latest` the setting finally started showing in logs. Yet topics are still read from the start :/ – Scourrge Jun 13 '22 at 13:45

1 Answers1

1

Not a fix but found this (currently open) issue that might be responsible: https://issues.apache.org/jira/browse/KAFKA-13988

Patrik
  • 21
  • 1