1

I want to run MirrorMaker as a standalone connector. So far I haven't found any documentation about the configuration. As far as I imagine the following would replicate myTopic.

Now in the destination cluster I need for the topic to have another name foo (not the automatic rename). Is this directly supported by MirrorSourceConnector or do I need some other means for that?

connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
tasksMax = 2
topics = myTopic
source.cluster.bootstrap.servers = sourceHost:9092
target.cluster.bootstrap.servers = sinkHost:9092
Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
abergmeier
  • 13,224
  • 13
  • 64
  • 120
  • Not sure. Can MirrorMaker1 be used as a Connector? – abergmeier May 15 '20 at 19:07
  • 1
    As of this date, the documentation on MirrorMaker2 is absolutely terrible. This video from Kafka Sumit 2020 indicates it is possible (around 14:35): https://www.confluent.io/resources/kafka-summit-2020/getting-up-to-speed-with-mirrormaker-2/ But I've no idea how to do it... – Robert Dec 02 '20 at 19:45

1 Answers1

4

So the Kafka Mirror Maker source code has a decent readme.md.

How you configure it is different depending on if you're running MM2 directly or in Kafka Connect. You said directly, which is in the linked readme.md.

Basically:

By default, replicated topics are renamed based on "source cluster aliases":

topic-1 --> source.topic-1

This can be customized by overriding the replication.policy.separator property (default is a period). If you need more control over how remote topics are defined, you can implement a custom ReplicationPolicy and override replication.policy.class (default is DefaultReplicationPolicy).

This unfortunately means that you cannot rename the topic through configuration code alone. (The DefaultReplicationPolicy class only allows you to specify the separator and nothing else). This is probably because when you specify the topics to mirror you use a regular expression, and not a single topic name (even if your source cluster topic config property is just the name of the topic - it's still treated like a regular expression).

So, back to the docs: ReplicationPolicy is a Java interface in the Kafka connect source code, so you would need to implement a custom Java class that implements ReplicationPolicy and then ensure it is on the classpath when you run MM2.

Let's imagine you do write such a class and you call it com.moffatt.kafka.connect.mirror.FooReplicationPolicy. A good template for your class is the default (and apparently only) replication policy class that comes with Kafka Connect: DefaultReplicationPolicy. You can see that building your own would not be too difficult. You could easily add a Map - either hard-coded or configured - that looks for specific configured topic names and maps it to the target topic name.

You use your new class by specifying it in the config as: replication.policy.class = com.moffatt.kafka.connect.mirror.FooReplicationPolicy

Robert
  • 348
  • 3
  • 11