5

We are using the mirror maker to sync on-premise and AWS Kafka topics. How can a topic with its schema registered in on-premise be replicated exactly the same in other clusters (AWS in this case)? How Avro schema is replicated using mirror maker?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Tony
  • 671
  • 1
  • 9
  • 29

1 Answers1

5

MirrorMaker only copies byte arrays, not schemas. And doesn't care about the format of the data

As of Confluent 4.x or later, Schema Registry added endpoint GET /schemas/ids/(number). So, if your consumers are configured to the original registry, this shouldn't matter since your destination consumers can lookup the schema ID.

You otherwise can mirror the _schemas topic as well, as recommend by Confluent when using Confluent Replicator

If you absolutely need one-to-one schema copying, you would need to implement a MessageHandler interface, and pass this on to the MirrorMaker command, to get and post the schema, similar to the internal logic I added to this Kafka Connect plugin (which you could use Connect instead of MirrorMaker). https://github.com/OneCricketeer/schema-registry-transfer-smt

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Small caveat, to "As of Confluent 4.x or later, this shouldn't matter", it doesn't apply to kafka-connect which still must get the schema version when deserializing https://github.com/confluentinc/schema-registry/blob/8b10c1c9571642bf39e7481b715ca966424cb6c6/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/AbstractKafkaProtobufDeserializer.java#L158 – 0x26res Sep 28 '22 at 17:07
  • I cannot remember why I wrote that 3 years, ago, but it was for MirrorMaker (ver. 1), and only primarily referencing Avro, not Protobuf – OneCricketeer Sep 28 '22 at 17:19