I'm having trouble using LocalDateTime in a Java Pojo and sending it through Kafka + Schema Registry using JsonSchema.
JsonSchema maps the LocalDateTime as an array of integers, but the actual message being sent through the stream contains a String which is causing an error when deserializing the message on the consumer side.
I also tried to create manually the schema, disabled auto.register.schemas and treat the date field as a String but it causes an error in the Producer side not being able to find the schema, because Schema Registry search the schema by the schema generated from the class not by and ID.
I created this project to reproduce the error https://github.com/alobatonavantica/kafka-poc
This is the pojo https://github.com/alobatonavantica/kafka-poc/blob/main/src/main/java/com/co/kafkapoc/dto/TestTypesDto.java
This is the Producer: https://github.com/alobatonavantica/kafka-poc/blob/main/src/main/java/com/co/kafkapoc/kafka/schemaregistry/KafkaSchemaRegistryProducer.java
Here are the configs of the Producer: https://github.com/alobatonavantica/kafka-poc/blob/e37766c60150debe798903ff698a8561ea2af5af/src/main/java/com/co/kafkapoc/config/KafkaConfig.java#L92
And this is the Consumer which is launched dynamically: https://github.com/alobatonavantica/kafka-poc/blob/main/src/main/java/com/co/kafkapoc/kafka/dynamic/KafkaDynamicConsumer.java
This is the error in the Consumer side when I use the schema auto registered by the producer:
org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.kafka.support.serializer.DeserializationException: failed to deserialize; nested exception is org.apache.kafka.common.errors.SerializationException: Error deserializing JSON message for id 304
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:2191) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.checkDeser(KafkaMessageListenerContainer.java:2202) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2107) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2039) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1967) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1853) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1543) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1190) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1087) [spring-kafka-2.6.8.jar:2.6.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_291]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_291]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_291]
Caused by: org.springframework.kafka.support.serializer.DeserializationException: failed to deserialize; nested exception is org.apache.kafka.common.errors.SerializationException: Error deserializing JSON message for id 304
at org.springframework.kafka.support.serializer.ErrorHandlingDeserializer.deserializationException(ErrorHandlingDeserializer.java:216) ~[spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.support.serializer.ErrorHandlingDeserializer.deserialize(ErrorHandlingDeserializer.java:191) ~[spring-kafka-2.6.8.jar:2.6.8]
at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:1365) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.access$3400(Fetcher.java:130) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1596) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) ~[kafka-clients-2.6.2.jar:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1283) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1174) [spring-kafka-2.6.8.jar:2.6.8]
... 4 common frames omitted
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing JSON message for id 304
Caused by: org.apache.kafka.common.errors.SerializationException: JSON {"stringNotRequired":null,"stringRequired":"test","booleanNotRequired":null,"booleanRequired":true,"numberNotRequired":null,"numberRequired":1.5,"integerNotRequired":null,"integerRequired":1,"arrayNotRequired":null,"arrayRequired":[1],"dateNotRequired":null,"dateRequired":"24-02-2022 04:28:58 PM"} does not match schema {"$schema":"http://json-schema.org/draft-07/schema#","title":"Test Types Dto","type":"object","additionalProperties":false,"properties":{"stringNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"string"}]},"stringRequired":{"type":"string"},"booleanNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"boolean"}]},"booleanRequired":{"type":"boolean"},"numberNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"number"}]},"numberRequired":{"type":"number"},"integerNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"integer"}]},"integerRequired":{"type":"integer"},"arrayNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"array","items":{"type":"integer"}}]},"arrayRequired":{"type":"array","items":{"type":"integer"}},"dateNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"array","items":{"type":"integer"}}]},"dateRequired":{"type":"array","items":{"type":"integer"}}},"required":["stringRequired","booleanRequired","numberRequired","integerRequired","arrayRequired","dateRequired"]}
Caused by: org.everit.json.schema.ValidationException: #/dateRequired: expected type: JSONArray, found: String
at org.everit.json.schema.ValidationException.copy(ValidationException.java:486) ~[org.everit.json.schema-1.12.2.jar:na]
at org.everit.json.schema.DefaultValidator.performValidation(Validator.java:67) ~[org.everit.json.schema-1.12.2.jar:na]
at org.everit.json.schema.Schema.validate(Schema.java:152) ~[org.everit.json.schema-1.12.2.jar:na]
at io.confluent.kafka.schemaregistry.json.JsonSchema.validate(JsonSchema.java:292) ~[kafka-json-schema-provider-6.2.0.jar:na]
at io.confluent.kafka.serializers.json.AbstractKafkaJsonSchemaDeserializer.deserialize(AbstractKafkaJsonSchemaDeserializer.java:133) ~[kafka-json-schema-serializer-6.2.0.jar:na]
at io.confluent.kafka.serializers.json.AbstractKafkaJsonSchemaDeserializer.deserialize(AbstractKafkaJsonSchemaDeserializer.java:88) ~[kafka-json-schema-serializer-6.2.0.jar:na]
at io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer.deserialize(KafkaJsonSchemaDeserializer.java:81) ~[kafka-json-schema-serializer-6.2.0.jar:na]
at org.apache.kafka.common.serialization.Deserializer.deserialize(Deserializer.java:60) ~[kafka-clients-2.6.2.jar:na]
at org.springframework.kafka.support.serializer.ErrorHandlingDeserializer.deserialize(ErrorHandlingDeserializer.java:188) ~[spring-kafka-2.6.8.jar:2.6.8]
at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:1365) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.access$3400(Fetcher.java:130) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1596) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1432) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:684) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:635) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1303) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237) ~[kafka-clients-2.6.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) ~[kafka-clients-2.6.2.jar:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1283) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1174) [spring-kafka-2.6.8.jar:2.6.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1087) [spring-kafka-2.6.8.jar:2.6.8]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_291]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_291]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_291
And this is the error in the Producer side when registering manually the schema using a string type in the date:
Error retrieving JSON schema: {"$schema":"http://json-schema.org/draft-07/schema#","title":"Test Types Dto","type":"object","additionalProperties":false,"properties":{"stringNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"string"}]},"stringRequired":{"type":"string"},"booleanNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"boolean"}]},"booleanRequired":{"type":"boolean"},"numberNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"number"}]},"numberRequired":{"type":"number"},"integerNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"integer"}]},"integerRequired":{"type":"integer"},"arrayNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"array","items":{"type":"integer"}}]},"arrayRequired":{"type":"array","items":{"type":"integer"}},"dateNotRequired":{"oneOf":[{"type":"null","title":"Not included"},{"type":"array","items":{"type":"integer"}}]},"dateRequired":{"type":"array","items":{"type":"integer"}}},"required":["stringRequired","booleanRequired","numberRequired","integerRequired","arrayRequired","dateRequired"]}
Any help or advice would be appreciated.
