1

After going through several lectures on schema registry and looking into how it works, I am more confused than before.

I would like to understand how can I include a schema registry in my kafka project where locally we have some producers and some consumers that deal with corresponding consumers/producers of a remote server.

If I understood correctly, the producer posts the schemaId of my avroFile (that has a version of this current schema) into schema registry and uploads some topic on the kafka queue with the schemaID in the payload header.

After that, the consumer will read from the queue, the topic that has the same schemaId (by invoking an API?) and the topic will be consumed.

Did I understand it right? Could you explain it to me, maybe with a diagram schema?

Thanks very much. Stefano

Bitswazsky
  • 4,242
  • 3
  • 29
  • 58
SteVizzo
  • 29
  • 5

1 Answers1

2

The overall flow, looks something like this: enter image description here While sending an avro payload, the Kafka Producer first registers the schema in the Schema Registry, if it's not already there. Then it prepares the avro byte-array by putting into it, the magic byte, schema ID (4 bytes) and the serialized avro bytes, in that order. A diagram of the actual payload, that goes to Kafka, is given below: enter image description here The consumer's job it the exact opposite. It validates the magic byte (which should always be zero, as of now), gets the schema ID, fetches the schema from Schema Registry using that ID, and finally uses that schema to deserialize the avro bytes. I'm omitting the serialization/deserialization part, as that's not mentioned in your question, and there are a lot of materials available online.

Bitswazsky
  • 4,242
  • 3
  • 29
  • 58
  • Hi, the problem is the really the deserialization step because usually we launch maven avro plugin for created automatically java classes used for deserializing the avro file of the producer but in this case how can i to generate automatically java deserializing classes for a different schema that i should to use retrieved by the Schema Registry API? – SteVizzo Aug 03 '18 at 08:11