2

I'd like to trigger a Dataflow job when new files are added to a Storage bucket in order to process and add new data into a BigQuery table. I see that Cloud Functions can be triggered by changes in the bucket, but I haven't found a way to start a Dataflow job using the gcloud node.js library.

Is there a way to do this using Cloud Functions or is there an alternative way of achieving the desired result (inserting new data to BigQuery when files are added to a Storage bucket)?

numentar
  • 1,049
  • 8
  • 21
  • 2
    There's an example of starting a Dataflow in this answer; does this help? http://stackoverflow.com/questions/35415868/launching-cloud-dataflow-from-cloud-functions – Sam McVeety Apr 01 '16 at 21:58
  • Thanks, this is useful indeed. I'm using the Dataflow Python SDK but hopefully that won't be an issue. – numentar Apr 15 '16 at 06:13
  • Please see my edited answer. – jkff Dec 19 '17 at 23:08

2 Answers2

2

This is supported in Apache Beam starting with 2.2. See Watching for new files matching a filepattern in Apache Beam.

jkff
  • 17,623
  • 5
  • 53
  • 85
  • have there been any new features in google cloud dataflow (beam) that make this process simpler in 2017? :) – jimmy Mar 27 '17 at 22:02
  • 1
    The mentioned JIRA issue - or more precisely, it's sub-issue BEAM-65, with the associated design https://s.apache.org/splittable-do-fn, has seen a lot of progress and it's my top priority right now. This particular feature is not implemented yet, but once the Splittable DoFn design itself works in all runners (it currently works in Direct runner, works-in-a-prototype in Dataflow streaming and Flink runners, and work is going on with Apex and Spark), it'll be among the first use cases we implement on top of it. – jkff Mar 28 '17 at 15:49
2

Maybe this post would help on how to trigger Dataflow pipelines from App Engine or Cloud Functions?

https://cloud.google.com/blog/big-data/2016/04/scheduling-dataflow-pipelines-using-app-engine-cron-service-or-cloud-functions

Mete Atamel
  • 821
  • 5
  • 11