I am quite new to using LinkedIn's camus and have successfully written data files from Kafka to Hdfs.
In general, I use JsonStringMessagdecoder
to read a JSON and write the same to .dat
file using StringRecordWriterProvider
.
But is it possible to write to multiple file types?
Suppose the json in kafka
is as follows :
{ "user":"John", "message":"Hi how are you?" }
Now I want John,Hi how are you?
to be written to one file, while user,message
to be written to another file(.meta
) in the same location. Is it possible ?