Apache Camel Kafka support for Confluent schema Registry

Question

I am trying to create camel route with kafka component trying to consume events with io.confluent.kafka.serializers.KafkaAvroDeserializer and schemaRegistry url along with other component parameters. I am not sure if this is full supported by Camel-Kafka currently. Can someone please comment on this ?

from("kafka:{{kafka.notification.topic}}?brokers={{kafka.notification.brokers}}"
                + "&maxPollRecords={{kafka.notification.maxPollRecords}}"
                + "&seekTo={{kafka.notification.seekTo}}"
                + "&specificAvroReader=" + "true"
                + "&valueDeserializer=" + "io.confluent.kafka.serializers.KafkaAvroDeserializer"
                +"&schemaRegistryURL=localhost:9021"
                + "&allowManualCommit={{kafka.notification.autocommit}})

specificAvroReader & schemaRegistryURL are the properties which seems to be not supported.

Refered https://github.com/apache/camel/blob/master/components/camel-kafka/src/main/docs/kafka-component.adoc. Seems like this is supposed to be a part of Camel 3.0.0-M2 release.But i dont see it in https://mvnrepository.com/artifact/org.apache.camel/camel-kafka or https://repository.apache.org/content/repositories/releases/org/apache/camel/apache-camel/ — user2109934, Mar 25 '19 at 15:58
Possible duplicate of [How to use camel-avro-consumer & producer?](https://stackoverflow.com/questions/55190052/how-to-use-camel-avro-consumer-producer) — OneCricketeer, Mar 29 '19 at 16:39

score 2 · Answer 1 · answered Mar 25 '19 at 17:20

I believe the only way currently to have camel-kafka to work with Confluent Schema Registry is to write a custom AvroSerilizer/ AvroDeserializer (io.confluent.kafka.serializers.AbstractKafkaAvroSerializer/ io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer). E.g.:

BlablaDeserializer extends AbstractKafkaAvroDeserializer implements Deserializer<Object>

and

BlablaSerializer extends AbstractKafkaAvroSerializer implements Serializer<Object>

and then set them on the camel component. E.g. for the value it will be:

 KafkaConfiguration kafkaConfiguration.setValueDeserializer(...)

i;m replying really very late, but it might help someone. yes you are right! and i have written customavro serilizer and deserializer https://stackoverflow.com/questions/55190052/how-to-use-camel-avro-consumer-producer/55234351#55234351 — driven_spider, Apr 22 '19 at 10:19

score 1 · Answer 2 · edited Sep 17 '19 at 13:18

1

Got this working after adding this to Gradle

compile 'org.apache.camel:camel-kafka:3.0.0-M2'

which can be found in this staging repository https://repository.apache.org/content/repositories/orgapachecamel-1124/org/apache/camel/

I think 3.0.0-M2 will be officially supported by Camel early next week.

Edit : 3.0.0-M2 available now https://repository.apache.org/content/repositories/releases/org/apache/camel/apache-camel/3.0.0-M2/

Has support for camel Kafka & Confluent Schema registry

edited Sep 17 '19 at 13:18

OneCricketeer

179,855
19
132
245

answered Mar 31 '19 at 04:08

user2109934

31
1
6

one of the links above doesn't work https://repository.apache.org/content/repositories/orgapachecamel-1124/org/apache/camel/. @cricket_007 directed me here but i couldn't see a solution. i had the exact same question but stackoverflow dint return me your page earlier. I have solved it a month ago now. if you did have a different solution, i would have loved to see without downloading the zip and taking that exercise – driven_spider Apr 22 '19 at 10:22
Is there a working example somewhere? I am losing my mind on this isssue – tstuber Oct 28 '20 at 16:05

score 0 · Answer 3 · edited Jan 04 '21 at 16:22

0

As already answered, this has been solved in the meantime. You do not have to write your own serializer/deserializer.

I made full example with camel-quarkus and schema-registry from confluent: https://github.com/tstuber/camel-quarkus-kafka-schema-registry

edited Jan 04 '21 at 16:22

Dharman

30,962
25
85
135

answered Jan 04 '21 at 16:16

tstuber

352
2
14

Apache Camel Kafka support for Confluent schema Registry

3 Answers3

Linked