Make Reactive Kafka work with Confluent Schema Registry Avro Schema

Question

How to make Reactive Kafka (https://github.com/akka/reactive-kafka) work with Confluent Schema Registry Avro Schema? Here is my sample code:

def create(groupId: String)(implicit system: ActorSystem): Source[ConsumerMessage.CommittableMessage[String, Array[Byte]], Consumer.Control] = {
    
    val consumerSettings = ConsumerSettings(system, new StringDeserializer, new ByteArrayDeserializer)
      .withBootstrapServers(bootstrap)
      .withGroupId(groupId)
      .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
      .withProperty("schema.registry.url", schemaRegistryUrl)
      .withProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, classOf[String].getName)
      .withProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, classOf[KafkaAvroDeserializer].getName)
      .withProperty(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true")
    
    Consumer.committableSource(consumerSettings, Subscriptions.topics(createDataSetJobRequestTopic))
}

score 1 · Answer 1 · answered Jun 19 '17 at 10:51

1

You just need to configure the consumer to use the Confluent deserializer : io.confluent.kafka.serializers.KafkaAvroDeserializer.class

Set the following properties :

ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG

In addition, you have to add the property "schema.registry.url" in order to specify at least one address pointing to your schema registry instance.

Finally, you have to add the following dependecy to your project :

            <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>${io.confluent.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Confluent Documentation

answered Jun 19 '17 at 10:51

fhussonnois

1,607
12
23

1

Thanks, I know how to consume kafka avro message with schema registry, the problem I asked is how to use it with reactive kafka. It requires to construct consumerSettings first like my pasted code above. I didn't know if it possible to get Source[ConsumerMessage.CommittableMessage[String, AvroClass] directly because KafkaAvroDeseriablizer is not generic. Otherwise I got CommittableMessage[String, Object] and then cast it to specific one? – Casel Chen Jun 19 '17 at 15:33
Hi, were you able to figure this out. I am having the exact problem of deserializing kafka messages through akka streams kafka. It returns an object which I am unable to cast to my type. – armulator Sep 18 '17 at 02:58
2

It seems the issue was not fixed and reported by others. https://github.com/akka/reactive-kafka/issues/342. – Casel Chen Dec 03 '17 at 12:42

Make Reactive Kafka work with Confluent Schema Registry Avro Schema

1 Answers1