Streaming writes to GCS using Apache Beam

Asked Jan 30 '18 at 23:39

Active Mar 15 '18 at 16:31

Viewed 771 times

I am using apache beam 2.2.0 and trying to write to GCS from my streaming pipeline once every 10 minutes using the following code:

input.apply(Window.<String>into(new GlobalWindows())
    .triggering(
            Repeatedly.forever(
                    AfterProcessingTime.pastFirstElementInPane()
                            .plusDelayOf(Duration.standardMinutes(10))
            ))
    .discardingFiredPanes())
    .apply("Write output files", TextIO.write()
            .to(baseuri + "/" + randomUUIDString())
            .withNumShards(1)
            .withSuffix(".csv")
            .withWindowedWrites());

I see the files getting written to the temporary directory (baseuri/UUIDString/.temp-beam-xxxxx) every 10 minutes but the writes never get finalized and written to baseuri + "/" + randomUUIDString(). Am I missing something here?

I have also attached the dataflow UI screenshot showing the "Write output files" step.

edited Jan 31 '18 at 10:42

Andrew Nguonly

2,258
1
17
23

asked Jan 30 '18 at 23:39

ghostcoder

Hrmm, according to https://stackoverflow.com/a/44032176/433352 global window with triggers should work. – ghostcoder Jan 31 '18 at 01:48
What version of beam are you using? Can you give a dataflow job id? – jkff Jan 31 '18 at 23:19
@jkff 2.2.0 version. Dataflow job is is 2018-01-30_15_45_08-11661064093628692248 . It's not running at the moment though. – ghostcoder Feb 01 '18 at 03:03
We're looking, thank you. – jkff Feb 01 '18 at 20:30
Any solution found here? – Daniel Messias Nov 19 '18 at 17:05

Streaming writes to GCS using Apache Beam

0 Answers0