Apache Camel with Kafka Schema registry

Question

I am building a Camel application to read message from Confluent Kafka. The messages are in Avro format and added below route configuration to read the Avro messages using schema registry in Camel route. When I enable the valueDeserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer, I am not getting any messages from Kafka topic. I tested the route with out schema registry and able to consume the message.

Route definition:

from("kafka:topic1?sslTruststoreLocation=<jks file>
&valueDeserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
&brokers=host1:9092,host2:9092,host3:9092
&sslKeystoreType=JKS
&groupId=grp1
&allowManualCommit=true
&consumersCount=10
&sslKeyPassword=<password>
&autoOffsetReset=earliest
&sslKeystorePassword=<password>
&securityProtocol=SSL
&sslTruststorePassword=<password>
&sslEndpointAlgorithm=HTTPS
&maxPollRecords=10
&sslTruststoreType=JKS
&sslKeystoreLocation=<keystore_path>
&autoCommitEnable=false
&additionalProperties.schema.registry.url=https://localhost:8081
&additionalProperties.basic.auth.user.info=abc:xyz
&additionalProperties.basic.auth.credentials.source=USER_INFO");

Can you please let me know, what is wrong in above configuration for schema registry. I also tried with EndPointRouteBuilder and same issue. However the producer application which is also Camel based and uses the schema registry for publishing Avro messages is working fine.

Your endpoint URI has the query parameter name `value-deserializer`, but it should be `valueDeserializer`. — Chin Huang, Feb 02 '22 at 16:11
I changed to valueDeserializer and ran again, but still same issue — PraveenKumar K G, Feb 02 '22 at 17:07

score 1 · Accepted Answer · answered Oct 04 '22 at 17:04

I figured out the way to configure the basic auth with Confluent schema registry. We need to configure as below

from("kafka:topic1?sslTruststoreLocation=<jks file>
&valueDeserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
&brokers=host1:9092,host2:9092,host3:9092
&sslKeystoreType=JKS
&groupId=grp1
&allowManualCommit=true
&consumersCount=10
&sslKeyPassword=<password>
&autoOffsetReset=earliest
&sslKeystorePassword=<password>
&securityProtocol=SSL
&sslTruststorePassword=<password>
&sslEndpointAlgorithm=HTTPS
&maxPollRecords=10
&sslTruststoreType=JKS
&sslKeystoreLocation=<keystore_path>
&autoCommitEnable=false
&additionalProperties.schema.registry.url=https://localhost:8081
&additional-properties[basic.auth.user.info]=abc:xyz
&additional-properties[basic.auth.credentials.source]=USER_INFO");

Note here, we need to use additional-properties for basic.auth.user.info and basic.auth.credentials.source as mentioned above.

score 0 · Answer 2 · answered Dec 05 '22 at 11:48

My issue was that the schema registry password contained special characters, such +.

So I had to wrap the property in RAW as described in the documentation [1]

Given the above example, it would then result in:

&additional-properties[basic.auth.user.info]=RAW(abc:xyz+)

[1] https://camel.apache.org/manual/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringparametervaluesusingrawvalues

Apache Camel with Kafka Schema registry

2 Answers2