Parquet Output From Kafka Connect to S3

Question

I see Kafka Connect can write to S3 in Avro or JSON formats. But there is no Parquet support. How hard would this be to add?

Added! See: https://twitter.com/karantasis/status/1181701302608285698?s=19 and: https://github.com/confluentinc/kafka-connect-storage-cloud/pull/241 — Aaron_ab, Oct 10 '19 at 08:43
Parquet support is now available as part of the 5.4 release of Kafka Connect S3 sink — Robin Moffatt, Jan 22 '20 at 11:55
Yes you can! I wrote an example [here](https://stackoverflow.com/a/73926873/9192415) — simple_developer, Oct 02 '22 at 17:27

score 5 · Accepted Answer · answered Jan 23 '20 at 15:16

5

Starting with Confluent 5.4.0, there is official support for Parquet output to S3.

answered Jan 23 '20 at 15:16

clay

18,138
28
107
192

score 2 · Answer 2 · answered May 11 '17 at 13:33

2

The Qubole connector supports writing out parquet - https://github.com/qubole/streamx

answered May 11 '17 at 13:33

diomedes01

106
1
6

score 1 · Answer 3 · answered Sep 23 '19 at 17:26

Try secor: https://github.com/pinterest/secor

Can work with AWS S3, google cloud, Azure's blob storage etc.

Note that the solution you choose must have key features like: Guarantee writing each message exactly once, load distribution, fault tolerance, monitoring, partitioning data etc.

Secor has it all and as stated above, can easily work with other "s3" style services..

Parquet Output From Kafka Connect to S3

3 Answers3

Linked