Kafka console consumer to read avro messages in HDP 3

Question

Trying to consume kafka Avro messages from console consumer and not exactly sure how to deserialize the messages.

sh /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server localhost:6667 --topic test --consumer.config /home/user/kafka.consumer.properties --from-beginning --value-deserializer ByteArrayDeserializer

The Avro Schema in Schema Registry for the test topic is:

{
 "type": "record",
 "namespace": "test",
 "name": "TestRecord",
 "fields": [
  {
   "name": "Name",
   "type": "string",
   "default": "null"
  },
  {
   "name": "Age",
   "type": "int",
   "default": -1
  }
 ]
}

Using HDP 3.1 version and Kafka-clients-2.0.0.3.1.0.0-78

Could someone help me what would be the Deserializer required to read Avro messages from console.

score 3 · Answer 1 · answered Apr 02 '20 at 15:31

3

Use kafka-avro-console-consumer

e.g.

sh /usr/hdp/current/kafka-broker/bin/kafka-avro-console-consumer.sh \
--bootstrap-server localhost:6667 \
--topic test \
--from-beginning \
--property schema.registry.url=http://localhost:8081

answered Apr 02 '20 at 15:31

Robin Moffatt

30,382
3
65
92

Unfortunately I don't have such script in bin folder. – William R Apr 02 '20 at 16:18
Try to locate it with `find /usr/hdp/current/ -name kafka-avro-console-consumer.sh` – Robin Moffatt Apr 02 '20 at 16:24
I tried already and i don't see such script available. Is it something i can download from public ? – William R Apr 02 '20 at 16:26
How does it work for HDP based Kafka distribution , I think Avro consumer is for confluent ? Because it fails with schema-registry-run-class missing after i copied the bash from public. – William R Apr 02 '20 at 16:34
Good point. In which case, head to https://www.confluent.io/download/ :) – Robin Moffatt Apr 02 '20 at 16:56
For this problem , Heading to confluent is not possible. i will try to figure out an alternate option. – William R Apr 02 '20 at 20:28
I'm have a schema registry address and the schema registered. Yet no decoding. – paiego Jul 18 '22 at 20:38

Kafka console consumer to read avro messages in HDP 3

1 Answers1