0

I'm trying to backup Kafka data using FileStreamSink connector. I know there are better options but my company already have file backup infrastructure (based on NetApp), so I'd like to dump Kafka data to a binary file and backup the file.

As the data stored in Kafka is encrypted, so we don't have a schema to use or transform. I tried to use this setting, but doesn't seem to work well: key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

Do you have suggestions for this case? Thanks.

superche
  • 753
  • 3
  • 9
  • 21

1 Answers1

1

See my answer at Kafka connect - string cannot be casted to struct

But, essentially, FileStreamSink can only write string values of records, and is not meant for production use, but rather as an example for writing your own sink connectors, which would then require you to implement a source connector to read that binary data back into a topic

If you don't want to implement your own Connector, then you'd need to implement some other consumer or look at mirroring your data to a secondary, backup Kafka cluster.

As mentioned elsewhere, backup of just a topic does not backup its configurations or any consumer groups associated with that topic

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245