Time limit possible for Google's Dataflow?

Asked Oct 30 '20 at 20:54

Active Nov 05 '20 at 16:49

Viewed 95 times

I've managed to use Google Cloud Scheduler to schedule a dataflow pipeline running, but I also want the pipeline to run for max an hour. Is it possible to schedule an end time for dataflow?

edit: I've created a pipeline that would wait a certain amount of time then cancel, but I'm getting the error on the cancel() line IOError: Failed to get the Dataflow job id.

Here's the pipeline code:

p = beam.Pipeline(options=PipelineOptions(region='us-central1'))

(p
    | 'ReadData' >> beam.io.ReadFromPubSub(topic=TOPIC).with_output_types(bytes)
    | 'Decode' >> beam.Map(lambda x:x.decode('utf-8'))
    | 'WriteToBigQuery' >> beam.io.WriteToBigQuery('{0}:MarkTest.scraped'.format(PROJECT), schema=schema, write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)
)
result = p.run()
result.wait_until_finish(duration=3000)

result.cancel()   # If the pipeline has not finished, you can cancel it

edited Nov 05 '20 at 16:49

asked Oct 30 '20 at 20:54

Mark Martinez

Do it a streaming pipeline? – guillaume blaquiere Oct 31 '20 at 12:51
1

Have you tried using `attempt_deadline`? Please provide a snippet of your code to have a better view on your function. – Alex G Nov 02 '20 at 06:29
Sure thing, adding it now! – Mark Martinez Nov 05 '20 at 16:47

Time limit possible for Google's Dataflow?

0 Answers0