1

I am new to flink I have transformation suppose like this 

val supportTask= customSource

  .map( line => line.split(","))
  .map( line => SupportTaskNew(line(0)toInt,line(1).toString,line(2)toString,line(3)toLong,line(4).toString,line(5)toInt,line(6)toInt))
  .filter(_ => true) //todo put sent date condition
  .map( line => Count(1))
  .keyBy(0)
  .timeWindow(Time.seconds(20)) //todo for time being 10 seconds, actuals 30 min
 .sum(0)

now I want to create file for each 20 seconds time window

supportTask.writeAsText(("D://myfile_"+Calendar.getInstance().get(Calendar.SECOND)),WriteMode.NO_OVERWRITE).setParallelism(1)

I have provided filename+seconds so that every time it create file with seconds appended to it.

But here only one file get created, I want to create new file for every 20 seconds window how I can do that ?

Sanjay Rabari
  • 2,091
  • 1
  • 17
  • 32
  • Use **DataStream.writeUsingOutputFormat() API**. **writeAsText** writes all output records to the file specified in the parameter. You have to implement a special output format to achieve that. – BrightFlow May 26 '17 at 02:16

1 Answers1

2

Perhaps you could do this with the Bucketing File Sink and a custom Bucketer.

David Anderson
  • 39,434
  • 4
  • 33
  • 60