Why kafka-avro-console-producer doesn't honour the default value for the field?

Question

Although default is defined for a field, kafka-avro-console-producer ignores it completely:

$ kafka-avro-console-producer --broker-list localhost:9092 --topic test-avro \
--property schema.registry.url=http://localhost:8081 --property \
value.schema='{"type":"record","name":"myrecord1","fields": \
[{"name":"f1","type":"string"},{"name": "f2", "type": "int", "default": 0}]}'

{"f1": "value1"}

org.apache.kafka.common.errors.SerializationException: Error 
deserializing json {"f1": "value1"} to Avro of schema 
{"type":"record","name":"myrecord1","fields": 
[{"name":"f1","type":"string"},{"name":"f2","type":"int","default":0}]}
Caused by: org.apache.avro.AvroTypeException: Expected int. Got END_OBJECT
    at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:698)
    at org.apache.avro.io.JsonDecoder.readInt(JsonDecoder.java:172)
    at org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83)
    at org.apache.avro.generic.GenericDatumReader.readInt(GenericDatumReader.java:511)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:182)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
    at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:240)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:230)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:174)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
    at io.confluent.kafka.formatter.AvroMessageReader.jsonToAvro(AvroMessageReader.java:213)
    at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:180)
    at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:54)
    at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

How to use it then in order to accept the default? Top level config is set to "BACKWARD" compatibility Level check, although I don't believe that has anything to do with the question. This schema is version 2, and version 1 was defined with only f1 field, but as I said, I don't think that matters.

AFAIK, the `default` value is only applicable to the reader/consumer. The field is still required to be defined by the writer/producer. — OneCricketeer, Apr 02 '19 at 21:38
I already voted for this as a useful comment, if you promote it to answer, I will accept it as the one that solves the problem, and is the most useful. — hdjur_jcv, Apr 03 '19 at 07:07
@hdjur_jcv I believe that my answer explains how to do what cricket_007 has described. — Giorgos Myrianthous, Apr 03 '19 at 07:12
Hi Giorgos Myrianthous, I have upvoted your answer as useful also, and I thank you for that, but I think that cricket_007's answer is the right one, because it reveals my misconception. Enabling null input value on producer side doesn't mean utilizing default zero value on consumer side (which should be the only my goal), as can be seen from your example. — hdjur_jcv, Apr 03 '19 at 07:41

OneCricketeer · Accepted Answer · 2022-03-07T16:45:13.740

4

As defined in the Avro spec

default: A default value for this field, used when reading instances that lack this field

Therefore, the producer still needs to supply that field.

I'm not sure it's possible to completely exclude a field when using the Avro console producer, because even if you make the field as nullable as Giorgos shows, you still need to explicitly set it.

edited Mar 07 '22 at 16:45

answered Apr 03 '19 at 19:01

OneCricketeer

179,855
19
132
245

score 2 · Answer 2 · answered Apr 02 '19 at 18:08

The error indicates that the message is not compatible with the Avro schema you've defined. As far as I understand, you want to allow null values for field f2. To do so, you need to change your value.schema to (note the definition of "type"):

value.schema='{"type":"record","name":"myrecord1","fields": [{"name":"f1","type":"string"},{"name": "f2", "type": ["null", "int"], "default": 0}]}'

but you would still need to define f2 key with null value. The following should do the trick for you:

kafka-avro-console-producer --broker-list localhost:9092 --topic test-avro \ 
    --property schema.registry.url=http://localhost:8081 \ 
    --property value.schema='{"type":"record","name":"myrecord1","fields": [{"name":"f1","type":"string"},{"name": "f2", "type": ["null", "int"], "default": 0}]}'

{"f1":"value1","f2":null}

And you can confirm that this has worked using kafka-avro-console-consumer:

kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic test-avro --from-beginning
{"f1":"value1","f2":null}
^CProcessed a total of 1 messages

Why kafka-avro-console-producer doesn't honour the default value for the field?

2 Answers2

Linked