1

I want to use apache avro schema's for data serialization and deserialization.

I want to use it with json encoding.

I want to put several of this serialized objects using different schemas to the same "source" (it's a kafka topic).

When I read it back I have the need to be able to resolve the right schema for the current data entry.

But the serialized data don't have any schema information in it. And to test all possible schema's for compatibility (kind of a duck typing approach) would be pretty unclean and error prone (for data which fits to multiple schemas it would be unclear which one to take)

I'm currently thought about putting the namespace and object name inside the json data programatically. But such a solution would not belong to the avro standard and it would open a new error scenario where it's possible to put the wrong schema namespace and/or object name inside the data.

I'm wondering if there would be a better way. Or if I have a general flaw in my solution.

Background: I want to use it for kafka message but don't want to use the schema registry (don't wan't to have a new single point of failure). I also still want to have KSQL support which is only available for json format or for using avro with the schema registry)

snap
  • 1,598
  • 1
  • 14
  • 21
  • You can have more than one instance of schema registry up and running, making it highly available. – Giorgos Myrianthous May 31 '19 at 13:06
  • Yes sure, but it have to be maintained in addition. And even if it's high available it's one more risk which is multiplied in the overall chain regarding the risk that the platform will not be available/work... – snap May 31 '19 at 13:48

0 Answers0