1

I am trying to use Confluent Kafka REST Proxy to retrieve data in Avro format from one of my topics but unfortunately I get a deserialization error. I am querying the Kafka REST proxy using the following command

 curl -X GET -H "Accept: application/vnd.kafka.avro.v2+json" 
http://localhost:8082/consumers/my-group/instances/my-consumer/records?timeout=30000

And I get as response 

{
  "error_code": 50002,
  "message": "Kafka error: Error deserializing key/value for partition input-0 at offset 0. If needed, please seek past the record to continue consumption."
}

and the logs on Kafka Rest Proxy server are:

org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition input-0 at offset 0. If needed, please seek past the record to continue consumption.
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

The data have been produced using KafkaAvroSerializer and the schema is present on the Schema Registry. Also note that data are readable by using avro-console-consumer on CLI.

Does anybody know how to resolve this issue?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
ypanag
  • 287
  • 5
  • 22

2 Answers2

0

It's most likely that as well as valid Avro messages on the topic, you also have invalid ones. That's what this error means, and is exactly the error that I got when I tried to consume a non-Avro message locally with the REST Proxy: 

ERROR Unexpected exception in consumer read task id=io.confluent.kafkarest.v2.KafkaConsumerReadTask@2e20d4f3  (io.confluent.kafkarest.v2.KafkaConsumerReadTask)
org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition avrotest-0 at offset 2. If needed, please seek past the record to continue consumption.
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

I would use a tool such as kafkacat to inspect the actual messages at the offset given in the error, e.g.: 

kafkacat -C -b localhost:9092 -t test_topic_avro -o 0 -c 1

The -o 0 will consume the message at offset 0, and -c 1 means consume just one message.

You can also seek past the problematic offset, e.g. for topic avrotest move the offset to 1: 

echo '{ "offsets": [ { "topic": "avrotest", "partition": 0, "offset": 1 } ] }' | \
http POST localhost:8082/consumers/rmoff_consumer_group/instances/rmoff_consumer_instance/positions \
Content-Type:application/vnd.kafka.v2+json
Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • 1
    Ok I can commit a message and move on to the next but the issue remains. On the other hand I can also read the messages using kafka-topics-ui. Could the issue be that I use different serializers for the key (StringSerializer) and the value (KafkaAvroSerializer) ? – ypanag Mar 12 '19 at 15:12
0

It wasn't supported to have String keys and AVRO values in the rest proxy until recently:

https://github.com/confluentinc/kafka-rest/issues/210

So recently that the code has been merged, but issue is still open and docs haven't been updated fully:

https://github.com/confluentinc/kafka-rest/pull/797

Ryan
  • 7,499
  • 9
  • 52
  • 61