1

We have bunch of avro files, we have schema first approach where kotlin producer project converts the file to class and produce the kafka message. Now there are multiple consumers which requires the same message.

I did see the reply here how to share avro schema definitions across teams , but somehow it does not feel very convenient to me.

We do have schema registry, but also in all consumers we want to use the same data model coming from the avro schema. So how do we share these schema file?

Have a different repo to store all the schemas? and then package and use it accross projects?

Or what other methods do people use?

SomeGuyWhoCodes
  • 579
  • 6
  • 19
  • What specific issues are you having with using the Registry? It's acts just as well as any HTTP endpoint where you can download the schemas during build time of the other services – OneCricketeer Nov 11 '21 at 00:16
  • You can also use Maven. Example - https://github.com/zolyfarkas/avro-schema-examples – OneCricketeer Nov 11 '21 at 00:22
  • Thanks. Specific issue is that if there are multiple consumers which are different micro service and different repo then we need a way to share the schema between these repository else each repository will have a copy of the same Avro schema or passed through as maven plug-in example you gave. We use the avro schema to convert it to Java class and then use it in our code. If we do not use the conversion in the consumer code then we will have to manually parse the generic record and convert it. Which is not nice either. – SomeGuyWhoCodes Nov 11 '21 at 05:09
  • You shouldn't be copying files manually. You could use Maven to publish the compiled classes (and schemas, if you want to) directly to a repo from the producer module (like link above); otherwise, using the registry maven plugin to download the schema w/ Avro plugin to generate the class works fine. Of course, if you need to access new fields that the producer adds, you'll need to stop and update any consumer code, regardless of what method you choose. That's not a unique problem to Kafka or the Schema Registry; for example you'd have the same issues with an HTTP or database client – OneCricketeer Nov 11 '21 at 12:44

0 Answers0