3

We are testing DR Scenario for kafka. we have 2 kafka cluster in separate region. We are using MirrorMaker2 to replicate the topics and messages. Topics and messages are able to replicate. But we are observing offset is not replicating.

e.g produced 10 messages from producuder pointed to kafka region 1.

Consumed 5 messages on from consumer pointed to kafka region 1

stop consumer pointed to region1

start consumer pointed to region2

consume the message

here expectation is region 2 consumer should consume from offset 6

but it starts consuming from offset 0

below is property file

 clusters = primary, secondary
# primary cluster information
 primary.bootstrap.servers = test1-primary.com:9094,test2-primary.com.apttuscloud.io:9094,test3-primary.com:9094
 primary.security.protocol= SASL_SSL
 primary.ssl.truststore.password= dummypassword
 primary.ssl.truststore.location= /opt/bitnami/kafka/config/certs/kafka.truststore.jks
 primary.ssl.keystore.password= dummypassword
 primary.ssl.keystore.location= /opt/bitnami/kafka/config/certs/kafka.keystore.jks
 primary.ssl.endpoint.identification.algorithm=
 primary.sasl.mechanism= PLAIN
 primary.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="dummyuser" password="dummypassword";

# secondary cluster information
 secondary.bootstrap.servers = test1-secondary.com:9094,test2-secondary.com.apttuscloud.io:9094,test3-secondary.com:9094
 secondary.security.protocol= SASL_SSL
 secondary.ssl.truststore.password= dummypassword
 secondary.ssl.truststore.location= /opt/bitnami/kafka/config/certs/kafka.truststore.jks
 secondary.ssl.keystore.password= dummypassword
 secondary.ssl.keystore.location= /opt/bitnami/kafka/config/certs/kafka.keystore.jks
 secondary.ssl.endpoint.identification.algorithm=
 secondary.sasl.mechanism=PLAIN
 secondary.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="dummyuser" password="dummypassword";
# Topic Configuration
 primary->secondary.enabled = true
 primary->secondary.topics = .*

 secondary->primary.enabled = true
 secondary->primary.topics = .*

############################# Internal Topic Settings  #############################
# The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and
# "mm2-offset-syncs.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3
 checkpoints.topic.replication.factor= 3
 heartbeats.topic.replication.factor= 3
 offset-syncs.topic.replication.factor= 3

# The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and
# "mm2-status.B.internal"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
 offset.storage.replication.factor=3
 status.storage.replication.factor=3
 config.storage.replication.factor=3

 replication.factor = 3
 refresh.topics.enabled = true
 sync.topic.configs.enabled = true
 refresh.topics.interval.seconds = 10
 topics.blacklist = .*[\-\.]internal, .*\.replica, __consumer_offsets
 groups.blacklist = console-consumer-.*, connect-.*, __.*
 primary->secondary.emit.heartbeats.enabled = true
 primary->secondary.emit.checkpoints.enabled = true

Please note some confedentilal values are placed with dummy values

Regards,

Narendra Jadhav

Narendra Jadhav
  • 41
  • 1
  • 4

1 Answers1

2

With MirrorMaker 2.5, when moving consumers between clusters, offsets are not automatically translated.

So upon starting consumers on another cluster, consumers need to use RemoteClusterUtils.translateOffsets() to find their offsets in this cluster.

In 2.7 (expected November 2020), you can have MirrorMaker 2 automatically translate offsets, see https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Hi @mickael-maison , RemoteClusterUtils seems to be only available for java based consumers , any way someone can achive it via other clients like .net ? some kind of configuration or steps which can be executed FYI, we are using Confluent.Kafka nuget Version 1.5.1 – Paras Patidar Sep 25 '20 at 04:42
  • RemoteClusterUtils is only available in Java. You can either wait for 2.7, or reimplement it in the language you need. The logic (https://github.com/apache/kafka/blob/trunk/connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/MirrorClient.java#L144-L183) is actually faily simple – Mickael Maison Sep 25 '20 at 08:29