0

I'm currently incapable of deserialize an avro PRIMITIVE key in a KSTREAM APP

the key in encoded with an avro schema ( registered in the schema registry ) ,

when i use the kafka-avro-console-consumer, I can see that the key is correctly deserialize

But impossible to make it work in a KSTREAM app

the avro schema of the key is a PRIMITIVE: 

{"type":"string"}

I already followed the documentation of confluent 

final Serde<V> valueSpecificAvroSerde = new SpecificAvroSerde<>();
final Map<String, String> serdeConfig = Collections.singletonMap(SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
valueSpecificAvroSerde.configure(serdeConfig, false);

final Serdes.StringSerde keySpecificAvroSerde = new Serdes.StringSerde();
keySpecificAvroSerde.configure(serdeConfig, true);

Consumed<String, totoAvro> inputConf = Consumed.with(keySpecificAvroSerde, valueSpecificAvroSerde);

final KStream<String, totoAvro> mystream = builder.stream("name topic", inputConf);

mystream.peek((key, value) -> logger.info("topic KEY :" + key))

it's working well for the value, but the key is going to be a string containing the bytes from the schema registry and not only the "reel" key

https://docs.confluent.io/current/schema-registry/serializer-formatter.html#wire-format

So the string key is /§/./11016015201 , but I would like the reel value : 1016015201

if I print the bytes inside the String it's [ 0x00 0x00 0x00 0x02 0x31 0x14 0x31 0x30 0x31 0x36 0x30 0x31 0x35 0x32 0x30 0x31 ]

raphaelauv
  • 670
  • 1
  • 11
  • 22
  • 3
    If you key if Avro, you need to use an AvroSerde, not `StringSerde` (`StringSerde`) is for plain String type only. – Matthias J. Sax Jul 26 '19 at 16:39
  • Hello, thank for your help. I'm sorry I should I said that I already tried with GenericAvroSerde ```java Exception caught during Deserialization, taskId: 6_3, topic: myTopic, partition: 3, offset: 0 (org.apache.kafka.streams.errors.LogAndFailExceptionHandler) java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.avro.generic.GenericRecord ``` It's a primitive avro schema , so it does not have a named field that I could get(). – raphaelauv Jul 29 '19 at 08:29
  • 1
    Primitive Avro types are not supported by Schema Registry. You will need to "hard code" the Avro Serde in your Streams code. – Matthias J. Sax Jul 29 '19 at 15:38
  • @MatthiasJ.Sax By primitive, do you mean non-record types? They are working fine for us ... `curl schema-registry:8081/subjects/topic-key/versions/1` shows `{"subject":"topic-key","version":1,"id":181,"schema":"\"string\""}` – OneCricketeer Aug 06 '19 at 22:20
  • Yes, I mean non-record types. Maybe you can register them, but you cannot "use" them. Cf. https://github.com/confluentinc/schema-registry/issues/1177 – Matthias J. Sax Aug 06 '19 at 23:51
  • 1
    The shipped Avro Serdes from schema registry always assume that it's a record-type. Hence, if you use a primitive type, you run into a class cast exception. – Matthias J. Sax Aug 06 '19 at 23:53

1 Answers1

0

Update

it's now working : https://stackoverflow.com/a/51957801/6227500

Original answer

The feature is not available currently in the schema registry project.

But by implementing a custom SERDE you can manage the case ,

Thiyaga Rajan proposed a working implementation

Serde class for AVRO primitive type

raphaelauv
  • 670
  • 1
  • 11
  • 22