0

I am trying to export Cloud Pub/Sub streams to Cloud Storage as described by this post by Spotify Reliable export of Cloud Pub/Sub streams to Cloud Storage or this post by Google Simple backup and replay of streaming events using Cloud Pub/Sub, Cloud Storage, and Cloud Dataflow

PubSub creates oubounded PCollection (or SCollection in SCIO) but saveastextfile requires BoundedCollection.

Is there any way to overcome this ?

Antoine Sauray
  • 241
  • 1
  • 3
  • 15

1 Answers1

1

The new dynamic IO module should support saving unbounded collection to files.

However note that the approach in that Spotify article doesn't use Dataflow since it has a lot of custom logic for SLA/bucketing/reliability reasons. So YMMV.

Neville Li
  • 420
  • 3
  • 10