How to handle Kafka schema evolution

Question

I'm new to kafka .Here is the question i have on ever changing kafka schema.

How can we handle schema changes at kafka consumer end?

If we change the payload structure at kafka publisher end, how can I make sure nothing breaks on the kafka consumer end?

I would be interested to know industry wide practises to handle this scenario.

I won't be using Confluent's schema registry for avro. Are there any other tried and tested options?

score 1 · Answer 1 · answered Apr 26 '20 at 06:31

1

Schema Registry is the solution for a centralized schema management and compatibility checks as schemas evolve Configure the schema registry into kafka producers and consumers

kafkaProducerProps.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG,"http://localhost:8081");

kafkaConsumerProps.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true");

//Schema registry location.
kafkaConsumerProps.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081");

Run Schema Registry on 8081

Refer below URL for sample code

https://dzone.com/articles/kafka-avro-serialization-and-the-schema-registry

answered Apr 26 '20 at 06:31

QuickSilver

3,915
2
13
29

As I mentioned , I dont want to use registry . – MonkTechie Apr 26 '20 at 20:45
You don't want to use any schema registry or its just the Confluent's schema registry ? – QuickSilver Apr 27 '20 at 03:59
Yes , no dependency on external server for retrieving schemas – MonkTechie May 01 '20 at 19:58

How to handle Kafka schema evolution

1 Answers1