Apache Kafka Avro Deserialization: Unable to deserialize or decode Specific type message.

Question

I am trying to use Avro Serialize with Apache kafka for serialize/deserialize messages. I am create one producer, which is used to serialize specific type message and send it to the queue. When message is send successfully to the queue, our consumer pick the message and trying to process, but while trying we are facing an exception, for case bytes to specific object. The exception is as below:

[error] (run-main-0) java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to com.harmeetsingh13.java.avroserializer.Customer
java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to com.harmeetsingh13.java.avroserializer.Customer
    at com.harmeetsingh13.java.consumers.avrodesrializer.AvroSpecificDeserializer.lambda$infiniteConsumer$0(AvroSpecificDeserializer.java:51)
    at java.lang.Iterable.forEach(Iterable.java:75)
    at com.harmeetsingh13.java.consumers.avrodesrializer.AvroSpecificDeserializer.infiniteConsumer(AvroSpecificDeserializer.java:46)
    at com.harmeetsingh13.java.consumers.avrodesrializer.AvroSpecificDeserializer.main(AvroSpecificDeserializer.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

According to exception, we are using some inconenient way for read the data, below is our code:

Kafka Producer Code:

static {
        kafkaProps.put("bootstrap.servers", "localhost:9092");
        kafkaProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
        kafkaProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
        kafkaProps.put("schema.registry.url", "http://localhost:8081");
        kafkaProducer = new KafkaProducer<>(kafkaProps);
    }


public static void main(String[] args) throws InterruptedException, IOException {
        Customer customer1 = new Customer(1002, "Jimmy");

        Parser parser = new Parser();
        Schema schema = parser.parse(AvroSpecificProducer.class
                .getClassLoader().getResourceAsStream("avro/customer.avsc"));

        SpecificDatumWriter<Customer> writer = new SpecificDatumWriter<>(schema);
        try(ByteArrayOutputStream os = new ByteArrayOutputStream()) {
            BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(os, null);
            writer.write(customer1, encoder);
            encoder.flush();

            byte[] avroBytes = os.toByteArray();

            ProducerRecord<String, byte[]> record1 = new ProducerRecord<>("CustomerSpecificCountry",
                    "Customer One 11 ", avroBytes
            );

            asyncSend(record1);
        }

        Thread.sleep(10000);
    }

Kafka Consumer Code:

static {
        kafkaProps.put("bootstrap.servers", "localhost:9092");
        kafkaProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
        kafkaProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
        kafkaProps.put(ConsumerConfig.GROUP_ID_CONFIG, "CustomerCountryGroup1");
        kafkaProps.put("schema.registry.url", "http://localhost:8081");
    }

    public static void infiniteConsumer() throws IOException {
        try(KafkaConsumer<String, byte[]> kafkaConsumer = new KafkaConsumer<>(kafkaProps)) {
            kafkaConsumer.subscribe(Arrays.asList("CustomerSpecificCountry"));

            while(true) {
                ConsumerRecords<String, byte[]> records = kafkaConsumer.poll(100);
                System.out.println("<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<" + records.count());

                Schema.Parser parser = new Schema.Parser();
                Schema schema = parser.parse(AvroSpecificDeserializer.class
                        .getClassLoader().getResourceAsStream("avro/customer.avsc"));

                records.forEach(record -> {
                    DatumReader<Customer> customerDatumReader = new SpecificDatumReader<>(schema);
                    BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(record.value(), null);
                    try {
                        System.out.println(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>");
                        Customer customer = customerDatumReader.read(null, binaryDecoder);
                        System.out.println(customer);
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });
            }

        }
    }

Using consumer in console, we are successfully able to receive the message. So what is the way for decode message into our pojo files ?

score 0 · Answer 1 · answered Feb 12 '17 at 14:23

The solution of this problem is, use

DatumReader<GenericRecord> customerDatumReader = new SpecificDatumReader<>(schema);

instead of

`DatumReader<Customer> customerDatumReader = new SpecificDatumReader<>(schema);

The exact reason for this, still not found. This may be, because Kafka, doesn't know about the structure of message, we explicitly define schema for message, and GenericRecord is useful to convert any message into readable JSON format according to schema. After creating JSON, we can easily convert it into our POJO class.

But Still, need to find solution for convert directly into our POJO class.

score 0 · Answer 2 · answered Feb 12 '17 at 14:26

0

You don't need to do the Avro serialization explicitly before passing the values to ProduceRecord. The serializer will do it for you. Your code would look like:

Customer customer1 = new Customer(1002, "Jimmy");
ProducerRecord<String, Customer> record1 = new ProducerRecord<>("CustomerSpecificCountry", customer1);
    asyncSend(record1);
}

See an example from Confluent for a simple producer using avro

answered Feb 12 '17 at 14:26

Javier Holguera

1,301
2
11
27

Hey @Javier, First think my question is related to Consumer not with Producer. Second: If you look into the example, the `JavaSessionize.avro.LogLine` is look like avro class, so may be they handle serialization for that. Third: I am using Specific-type conversion not generic conversion. Avro only support 8 types, otherwise we need to define the whole schema conversion. – Harmeet Singh Taara Feb 12 '17 at 14:37
@HarmeetSinghTaara I suggested looking into the producer because, if you serialise something different to what you expect, the consumer will fail. I'd recommend to use Confluent's kafka-avro-console-consumer to read the event and see if it looks like what you expect. If that fails as well, then the problem is not in your consumer. – Javier Holguera Feb 12 '17 at 15:06
@HarmeetSinghTaara `LogLine` is the result of passing [this avro schema](https://github.com/confluentinc/examples/blob/3.1.x/kafka-clients/specific-avro-producer/src/main/resources/avro/LogLine.avsc) through the `maven-avro-plugin` that generates Java classes automatically. If you compile the project you can have a look and compare with your own `Customer` class. I don't think they do any serialization inside `LogLine`, I'm pretty sure it's just a POJO. – Javier Holguera Feb 12 '17 at 15:09
I am generating Customer class with sbt plugin. Look into this: https://github.com/sbt/sbt-avro – Harmeet Singh Taara Feb 12 '17 at 15:17
Hey @Javier, I am trying to follow avro example but getting exception. Please look into http://stackoverflow.com/questions/42200875/apache-kafka-and-avro-org-apache-avro-generic-genericdatarecord-cannot-be-cast – Harmeet Singh Taara Feb 13 '17 at 09:39

Apache Kafka Avro Deserialization: Unable to deserialize or decode Specific type message.

2 Answers2