1

In our restAPI we will get a complex JSON payload and map it into a POJO. And based on the avro avsc schema file I use avro-maven-plugin to generate some avro schema class.

My question is when we send message to kafka and schema registry by using KafkaTemplate, we need to send with avro schema object. We can't manually map values from the payload request object into the avro schema object due to the huge number of fields.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Kuku
  • 484
  • 8
  • 28
  • Hi Kuku, Kindly check my answer https://stackoverflow.com/a/71656174/12894468 . We can convert any POJO classes to Avro records. If you want more info, let me know – Sivaram Rasathurai May 18 '22 at 03:57

1 Answers1

1

Two steps to convert any pojo class to avro genric record

  1. Using jackson/avro, to convert the pojo into bytes with Avro Mapper.

  2. Using Avro GenericDatumReader to read it as Generic Record.

public class AvroConverter{

 public static GenericRecord convertToGenericRecord(String schemaPath, SomeClass someObject){
  Schema schema = new Schema.Parser().setValidate(true).parse(new ClassPathResource(schemaPath).getFile());
  AvroSchema avSchema = new AvroSchema(schema);
  ObjectWritter writter = new AvroMapper().writer(avSchema);
  final byte[] bytes = objectWriter.writeValueAsBytes(someObject);
  GenericDatumReader<Object> genericRecordReader = new GenericDatumReader<>(avSchema);
  return (GenericRecord) genericRecordReader.read(null, DecoderFactory.get().binaryDecoder(bytes, null));
 }

}

Gradle Dependency

    implementation 'com.fasterxml.jackson.dataformat:jackson-dataformat-avro'

While doing serialization, you may face issues. For that, you have to configure the avro mapper properties

Sivaram Rasathurai
  • 5,533
  • 3
  • 22
  • 45
  • Have you faced with the issue when there is some enum type in the avro schema file. It throws JsonMappingException: Not an enum during the call objectWriter.writeValueAsBytes ? – Kuku May 30 '22 at 11:02
  • 1
    I didn't face that. We sent as String if we have enum. basically the validation will happens before converting to string – Sivaram Rasathurai May 30 '22 at 15:20
  • I think you are right, the validation will happen when the request comes in. The IllegalArgumentException will be thrown and could be handled globally. Thanks – Kuku May 31 '22 at 10:38
  • 1
    Are GenericRecord really required? You could use ReflectData instead and you'd then have a more concrete type – OneCricketeer Jun 04 '22 at 14:32
  • @RCvaram Did you face with memory leak issue for this approach? I notice there are some memory leak after deployed into prodction – Kuku Aug 26 '22 at 07:39