Is it possible to replicate kafka topics without alias prefix with MirrorMaker2

Question

I'm trying to set up replication between 2 clusters but don't want want the topic names to be changed. for example if i have a topic called "some_topic" it is automatically replicated to "cluster1.some_topic", I'm pretty sure this can be done but haven't found the correct config to change this

My current config "mirrormaker2.properties"

# Sample MirrorMaker 2.0 top-level configuration file
# Run with ./bin/connect-mirror-maker.sh connect-mirror-maker.properties 

# specify any number of cluster aliases
clusters = cluster1, cluster2

# connection information for each cluster
cluster1.bootstrap.servers = host1:9092,host2:9092,host3:9092
cluster2.bootstrap.servers = rep_host1:9092,rep_host2:9092,rep_host3:9092

# enable and configure individual replication flows
cluster1->cluster2.enabled = true
cluster1->cluster2.topics = sometopic.*

# customize as needed
# replication.policy.separator = _
# sync.topic.acls.enabled = false
# emit.heartbeats.interval.seconds = 5

for reference:

score 13 · Answer 1 · answered Mar 10 '20 at 14:03

To 'disable' topic prefixes and to have topic properties mirrored properly at the same time, I had to provide a customized replication policy which also overrides the topicSource method. Otherwise non-default topic properties (e.g., "cleanup.policy=compact") have not been mirrored, even after restarting mirror maker.

Here is the complete procedure that worked for me:

Compile and package the following customized replication policy into a .jar file (full source code can be found here):

public class PrefixlessReplicationPolicy extends DefaultReplicationPolicy {

  private static final Logger log = LoggerFactory.getLogger(PrefixlessReplicationPolicy.class);

  private String sourceClusterAlias;

  @Override
  public void configure(Map<String, ?> props) {
    super.configure(props);
    sourceClusterAlias = (String) props.get(MirrorConnectorConfig.SOURCE_CLUSTER_ALIAS);
    if (sourceClusterAlias == null) {
      String logMessage = String.format("Property %s not found", MirrorConnectorConfig.SOURCE_CLUSTER_ALIAS);
      log.error(logMessage);
      throw new RuntimeException(logMessage);
    }
  }

  @Override
  public String formatRemoteTopic(String sourceClusterAlias, String topic) {
    return topic;
  }

  @Override
  public String topicSource(String topic) {
    return topic == null ? null : sourceClusterAlias;
  }

  @Override
  public String upstreamTopic(String topic) {
    return null;
  }
}

Copy the .jar into the ${KAFKA_HOME/libs directory
Configure Mirror Maker 2 to use that replication policy by setting the replication.policy.class property in ${KAFKA_HOME}/config/mm2.properties:

  replication.policy.class=ch.mawileo.kafka.mm2.PrefixlessReplicationPolicy

This is a pretty good answer for me but I got one question : why didn't you just change the method formatRemoteTopic and call `super` on other method ? (worked on my side) — Naremy, Mar 12 '20 at 16:50
Hi @Naremy, by 'other method' do you mean `topicSource` and `upstreamTopic`? When I left those methods unchanged then mirroring of topics' properties did not work. In my Kafka setup for example topics by default are not compacted; and when I created a topic which I wanted to be compacted (so I set the `"cleanup.policy=compact"` property for that topic) the mirrored topic on the downstream side still had the default properties (i.e., no compaction). Does mirroring of topic properties work with your solution? — Marcin Wieloch, Mar 17 '20 at 07:19
In the future it looks like this will not be necessary. As of July 2021 a IdentityReplicationPolicy has been added to MirrorMaker2. It does not appear to be part of a release yet. https://github.com/apache/kafka/blob/trunk/connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/IdentityReplicationPolicy.java — hayduke, Sep 27 '21 at 21:26

score 11 · Answer 2 · answered Feb 05 '22 at 00:20

11

Starting with Kafka 3.0.0, it is sufficient to set

replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy

Also, the PrefixlessReplicationPolicy in marcin-wieloch's answer https://stackoverflow.com/a/60619233/12008693 no longer works with 3.0.0 (NullPointerException).

answered Feb 05 '22 at 00:20

Gunther Vogel

133
1
7

When I use this policy, would it overwrite the data in the Target topic? Given that Target topic already exists and has data. – flowAlong Mar 30 '23 at 11:36

score 10 · Answer 3 · answered Jan 20 '20 at 11:26

I was able to remove the prefix using this setup:

"replication.policy.separator": ""
"source.cluster.alias": "",
"target.cluster.alias": "",

If the alias setting is necessary in your case, I understand you should use other replicationPolicy class. By default is using DefaultReplicationPolicy class (https://kafka.apache.org/24/javadoc/org/apache/kafka/connect/mirror/DefaultReplicationPolicy.html)

score 6 · Accepted Answer · edited Feb 17 '20 at 09:51

6

I think the answer above is inappropriate.

In Mirror Maker 2.0, if you want to keep the topic unmodified, you have to implement ReplicationPolicy.

You can refer to DefaultReplicationPolicy.class, and then override formatRemoteTopic(), after that you have to remove sourceClusterAlias + separator. In the end, configure replication.policy.class in the mm2.properties

I defined MigrationReplicationPolicy.class

replication.policy.class = org.apache.kafka.connect.mirror.MigrationReplicationPolicy

You should see MirrorClientConfig,class, I know that you will understand

edited Feb 17 '20 at 09:51

Matteo Ragni

2,837
1
20
34

answered Feb 17 '20 at 06:37

Zhou Yang

99
1
3

3

Can you post a link to the custom `MigrationReplicationPolicy.class` you created ? – tortuga May 11 '20 at 20:58

score 4 · Answer 5 · answered Jun 11 '20 at 08:14

Managed to push replication with Kafka ConfluentINC Connector image release 5.4.2 Properties are:

connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector
target.cluster.alias= 
replication.factor=3
tasks.max=3
topics=.*
source.cluster.alias= 
target.cluster.bootstrap.servers=<broker1>,<broker2>,<broker3>
replication.policy.separator= 
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
source.cluster.bootstrap.servers=<broker1>,<broker2>,<broker3>
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter

1) Leave space after the 3 parameters: source.cluster.alias, replication.policy.separator, target.cluster.alias.

2) Set this mirror connector on the TARGET Kafka, NOT on the source (performs pull only)

In addition yo can use Conductor or Kafka Connector UI landoop image - landoop/kafka-connect-ui

This is still in a testing scenario, but it looks promising.

score 3 · Answer 6 · edited Sep 07 '21 at 10:48

I'm trying to set up replication between 2 clusters, but need the same topic name in both the cluster without giving an alias for the in the connect-mirror-maker.properties.

By default, replicated topics are renamed based on source cluster aliases.

    Source --> Target
    topic-1 --> source.topic-1

You can avoid topics being renamed by setting the following properties to blank under your connector properties file. By default, replication.policy.separator property is a period, then by setting it to blank along with the source.cluster.alias, the target topic will have the same name as the source topic.

replication.policy.separator=
source.cluster.alias=
target.cluster.alias=

Is it possible to replicate kafka topics without alias prefix with MirrorMaker2

6 Answers6

Linked