We are using the mirror maker to sync on-premise and AWS Kafka topics. How can a topic with its schema registered in on-premise be replicated exactly the same in other clusters (AWS in this case)? How Avro schema is replicated using mirror maker?
1 Answers
MirrorMaker only copies byte arrays, not schemas. And doesn't care about the format of the data
As of Confluent 4.x or later, Schema Registry added endpoint GET /schemas/ids/(number)
. So, if your consumers are configured to the original registry, this shouldn't matter since your destination consumers can lookup the schema ID.
You otherwise can mirror the _schemas
topic as well, as recommend by Confluent when using Confluent Replicator
If you absolutely need one-to-one schema copying, you would need to implement a MessageHandler
interface, and pass this on to the MirrorMaker command, to get and post the schema, similar to the internal logic I added to this Kafka Connect plugin (which you could use Connect instead of MirrorMaker). https://github.com/OneCricketeer/schema-registry-transfer-smt

- 179,855
- 19
- 132
- 245
-
Small caveat, to "As of Confluent 4.x or later, this shouldn't matter", it doesn't apply to kafka-connect which still must get the schema version when deserializing https://github.com/confluentinc/schema-registry/blob/8b10c1c9571642bf39e7481b715ca966424cb6c6/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/AbstractKafkaProtobufDeserializer.java#L158 – 0x26res Sep 28 '22 at 17:07
-
I cannot remember why I wrote that 3 years, ago, but it was for MirrorMaker (ver. 1), and only primarily referencing Avro, not Protobuf – OneCricketeer Sep 28 '22 at 17:19