0

I would like to consume data from pubsub through dataflow streaming job and store it into GCS in hourly directories.

What would be best approach?

I tried using WindowedFilenamePolicy but it adds an additional group by and slows down the write operation at the time of writes. Dataflow buffers the data correctly but takes too long to write data in temp bucket.

Any best practice for such fairly common case?

Regards, Pari

Maxim
  • 4,075
  • 1
  • 14
  • 23
Pari
  • 1,443
  • 3
  • 19
  • 34

1 Answers1

2

Using the Google-Provided Dataflow Template for the streaming pipeline from Cloud Pub/Sub to Google Cloud Storage files, you can easily do it by setting the outputDirectory to gs://<BUCKET>/YYYY/MM/DD/HH/ and it will automatically replace YYYY, MM, DD and HH for the values of the interval window.

Héctor Neri
  • 1,384
  • 9
  • 13
  • Thanks Neri. I am using the very similar code but it is slowing down the writes to GCS. – Pari Sep 08 '18 at 07:16
  • This template has a window of 5 minutes by default so it creates a file each 5 minutes, you can use the example usage provided in its [source code](https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubsubToText.java) and adjust the window duration to its minimum of 1s. – Héctor Neri Sep 10 '18 at 14:45