2

When we were trying to write the processed JSON data from Flink into file using the method writeAsCsv(path, writemode), the data as written into the file but we need to insert comma after every JSON which is not happening. We are using Apache Kafka as data source to Flink.

DataStream<Tuple5<String, String, String, String, String>> messageStream = env.addSource(new FlinkKafkaConsumer08<>(FLINK_TOPIC, new SimpleStringSchema(), properties)).flatMap(new StreamToTuple5()).keyBy(0);
String path = "/home/user/Documents/docs/csvfile/";
messageStream.writeAsCsv(path, FileSystem.WriteMode.OVERWRITE);

the output obtaining on executing this method is 

{"temperSensorData":"28.489084691371364","temperSensorUnit":"celsius","timestamp":"1493270680759","timestamp2":"1493270680786","timestamp3":"1493270680787"}
{"temperSensorData":"28.489084691371467","temperSensorUnit":"celsius","timestamp":"1493270680761","timestamp2":"1493270680816","timestamp3":"1493270680816"}
Till Rohrmann
  • 13,148
  • 1
  • 25
  • 51
Arch
  • 73
  • 1
  • 8

1 Answers1

0

You can solve your problem by specifying ',' as your row delimiter:

messageStream.writeAsCsv(path, FileSystem.WriteMode.OVERWRITE, ",", ",")

Here the third argument is the row delimiter.

Till Rohrmann
  • 13,148
  • 1
  • 25
  • 51