Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-mirrormaker]

MirrorMaker is a Kafka tool to mirror a source Kafka cluster into a target (mirror) Kafka cluster. MirrorMaker2 is a newer implementation of the tool, and is based on the Kafka Connect Framework.

168 questions
15
votes
6 answers

Is it possible to replicate kafka topics without alias prefix with MirrorMaker2

I'm trying to set up replication between 2 clusters but don't want want the topic names to be changed. for example if i have a topic called "some_topic" it is automatically replicated to "cluster1.some_topic", I'm pretty sure this can be done but…
Brendan Scullion
  • 390
  • 1
  • 3
  • 16
9
votes
1 answer

Disable mirrormaker2 offset-sync topics on source kafka cluster

We're using MirrorMaker2 to replicate some topics from one kerberized kafka cluster to another kafka cluster (strictly unidirectional). We don't control the source kafka cluster and we're given only access to describe and read specific topics that…
Alexandre Juma
  • 3,128
  • 1
  • 20
  • 46
8
votes
4 answers

Kafka MirrorMaker2 - not mirroring consumer group offsets

I have setup MirrorMaker2 for replicating data between 2 DCs. My mm2.properties: # mm2.properties name=source->dest clusters=source,…
user1578872
  • 7,808
  • 29
  • 108
  • 206
6
votes
1 answer

Kafka MirrorMaker2 automated consumer offset sync

I am using MirrorMaker2 for DR. Kafka 2.7 should support automated consumer offset sync Here is the yaml file I am using (I use strimzi for creating it) All source cluster topics are replicated in destination cluster. Also ...checkpoint.internal…
Inako
  • 259
  • 5
  • 13
6
votes
1 answer

How customer offsets are maintained in mirrored cluster in Kafka?

Lets say I have two Kafka clusters and I am using mirror maker to mirror the topics from one cluster to another. I understand consumer has an embedded producer to commit offsets to __consumer-offset topic in Kafka cluster. I need to know what will…
Yogesh Gupta
  • 1,226
  • 1
  • 12
  • 23
5
votes
1 answer

Mirrormaker2.0 vs confluent replicator

Apache Kafka introduced Mirrormaker2(MM2) with the release of Kafka 2.4. MM2 is significantly better than MM1. I know that from the architecture standpoint, MM1 used to work using producer and consumer API whereas MM2 uses connect API. I believe…
JR ibkr
  • 869
  • 7
  • 24
5
votes
2 answers

Kafka MirrorMaker 2.0 duplicate each messages

I am trying to replicate Kafka cluster with MirrorMaker 2.0. I am using following mm2.properties: name = mirror-site1-site2 topics = .* connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector tasks.max =…
Aurelien
  • 51
  • 1
  • 2
5
votes
1 answer

How to replicate schema with Kafka mirror maker?

We are using the mirror maker to sync on-premise and AWS Kafka topics. How can a topic with its schema registered in on-premise be replicated exactly the same in other clusters (AWS in this case)? How Avro schema is replicated using mirror maker?
Tony
  • 671
  • 1
  • 9
  • 29
4
votes
1 answer

kafka mirrormaker 2 custom partitioner

I am trying to determine if it is possible to use a custom partitioner with mirrormaker 2 so when replicating to the target cluster my custom partitioner is used. According to the docs here https://github.com/apache/kafka/tree/trunk/connect/mirror…
S P
  • 71
  • 5
4
votes
2 answers

Setting up Kafka HA with MirrorMaker - how to handle that in Flink?

We are setting up MirrorMaker 2.0 for Kafka. If I understand correctly, topic offsets are not equal in replicated cluster. This is not a problem for regular Kafka app as consumer groups get replicated too. Flink stores Kafka offsets internally in…
bottaio
  • 4,963
  • 3
  • 19
  • 43
3
votes
1 answer

Mirrormaker 2: different partition count

I am using MM2 to replicate clusters. The current topics are overprovisioned (too many partitions) and I need to reduce them as part of the migration to fit actual needs & confluent cloud restrictions. Is this at all possible? I see that this helps…
V-Lamp
  • 1,630
  • 10
  • 18
3
votes
0 answers

Re-sync offsets with MirrorMaker2

MirrorMaker2 (strimzi:0.22.1-kafka-2.7.0) is configured to do one directional mirroring from broker A to B with group offset sync enabled: sync.group.offsets.enabled: true sync.group.offsets.interval.seconds: 30 The setup was working fine to a…
m3nthal
  • 413
  • 5
  • 14
3
votes
2 answers

How to migrate consumer offsets using MirrorMaker 2.0?

With Kafka 2.7.0, I am using MirroMaker 2.0 as a Kafka-connect connector to replicate all the topics from the primary Kafka cluster to the backup cluster. All the topics are being replicated perfectly except __consumer_offsets. Below are the connect…
Amit Yadav
  • 4,422
  • 5
  • 34
  • 79
3
votes
1 answer

Mirrored repartition topics keep increasing in size

We are using MirrorMaker to backup topics. We noticed that repartition topics created by Kafka Streams applications seem to keep increasing in size in the target cluster compared to the source cluster. This makes sense according to the documentation…
Martin Kamp Jensen
  • 157
  • 3
  • 10
3
votes
1 answer

Kafka Mirror Maker 2 Offset Replication Not Working

We are testing DR Scenario for kafka. we have 2 kafka cluster in separate region. We are using MirrorMaker2 to replicate the topics and messages. Topics and messages are able to replicate. But we are observing offset is not replicating. e.g produced…
Narendra Jadhav
  • 41
  • 1
  • 4
1
2 3
…
11 12