0

I'm relatively new to Avro but I'm fairly certain that I've read and experimented enough to understand how it generally works.

I'm working on a Java distributed system where the Avro schemas are defined by a single consumer. The schemas are used to generate the POJOs which are then imported by the producer applications. Due to variable release cadence of our system, different producer applications can be generating messages using old versions of the schema - this is why we chose Avro.

Due to the way we define and generate the schemas in a single Maven module, there's no real need for us to deploy a full-scale registry. We have no use-case where we need to dynamically look up a schema for either serialisation or deserialisation.

All we actually need is a strategy (and tooling) which will allow us to write an implementation of SchemaStore which is backed by schema files on the classpath. The module can back bundled into a Maven package and imported into the producer apps.

Is anyone aware of any methods or tooling which allow us to manage multiple versions of schema files, track version numbers, and automatically generate the POJO classes?

I'm imagining something that works a bit like the old Maven version plugin. We have the "current" version of the schema committed in code. When we want to make a change, we change this file and then run the plugin which automatically commits it as a new version and generates the new POJO. The old version is maintained and also committed in code so that it remains available to the SchemaStore.

It's worth noting that we're not using Kafka or any other kind of streaming system. All messages are sent using plain-old REST endpoints and are serialised using single-object-encoding (so they contain the schema fingerprint)

I hope this all makes sense. Any help would be much appreciated!

icecreamhead
  • 111
  • 1
  • 12
  • Does this help? https://github.com/zolyfarkas/avro-schema-examples – OneCricketeer Dec 02 '20 at 17:06
  • Looks promising. I'll have a look through and let you know, thanks! – icecreamhead Dec 04 '20 at 15:58
  • I think your project is along the right lines, but I was hoping for something a little simpler that I could just re-use. – icecreamhead Dec 07 '20 at 12:58
  • Not my project, but there's also the Hortonworks Schema Registry and a registry project by Apicurio on Github that aren't tied to Kafka nor Maven – OneCricketeer Dec 07 '20 at 14:48
  • Unfortunately both of these still need a hosted database of some kind. I'm trying to implement a schema store using only jars on the classpath. – icecreamhead Dec 11 '20 at 15:25
  • Okay... You could package a jar that creates an in memory database of schemas, or stores schema files as resources. Wouldn't be much different than a standalone project doing the same thing – OneCricketeer Dec 11 '20 at 15:28
  • Yeah that's exactly my objective - package the schema files as resources. The bit I don't yet have is the tooling to automatically version them. (I can do this by hand of course, but I'd like to automate it as there will be a number of contributors in future.) – icecreamhead Dec 14 '20 at 12:27
  • I suppose you could write your own maven plugin, but I'm not sure how you'd tell it that you have a new version available – OneCricketeer Dec 14 '20 at 14:54
  • 1
    Update: we ended up writing our own tooling. We've not bundled it into a maven plugin yet, but we probably will. – icecreamhead Feb 16 '21 at 16:24

0 Answers0