I'm using Kafka connect with 2 connectors:
- debezium to pull data from Postgres to Kafka
- S3 connector to save data from Kafka to S3
While running I got this error from the S3 connector
java.lang.NullPointerException: Array contains a null element at 0
I have found the related message that has the following as part of the message:
"some_key": [
"XCVB",
null
]
How can I process this message?
I have tried adding the following to the S3 connector config:
"behavior.on.null.values": "ignore",
"errors.tolerance": "all",
"errors.deadletterqueue.topic.name":"dlq_s3_sink"
to try and skip those messages and send them to DLQ, but it doesn't seems to be working and the task failed on this error. I also saw this in the log:
Set parquet.avro.write-old-list-structure=false to turn on support for arrays with null elements.
but not sure where should I add this? as part of the connector config?