How to append kafka consumer output to a file(parquet) in HDFS using Java/Scala?

Question

This is a Kafka batch process. I want to read a local CSV file and write it into a Kafka topic.

Then consumer has to get data from the topic they subscribed.

Expected: I want the consumed data to be appended to a file in Parquet format in HDFS. Please help me to achieve this in an efficient manner.

Kafka Producer input:

Kafka Consumer output:

I want the value to be appended to a file in HDFS.

Does this answer your question? [create parquet files in java](https://stackoverflow.com/questions/39728854/create-parquet-files-in-java) — mazaneicha, Mar 05 '20 at 13:59

score 0 · Answer 1 · answered Mar 05 '20 at 13:45

Doing that from scratch would be quite complicated.

You can use the Kafka connect sink HDFS connector that handle out of the box parquet format output ( this would need a bit pre processing your records though, to put them in a correct format like json with schema etc..).

More info here : https://docs.confluent.io/current/connect/kafka-connect-hdfs/index.html

How to append kafka consumer output to a file(parquet) in HDFS using Java/Scala?

1 Answers1