4

I'm trying to run a python script in Google Cloud which will download 50GB of data once a day to a storage bucket. That download might take longer than the timeout limit on the Google Cloud Functions which is set to 9 minutes. The request to invoke the python function is triggered by HTTP.

Is there a way around this problem ? I don't need to run a HTTP Restful service as this is called once a day from an external source. (Can't be scheduled) .

The whole premise is do download the big chuck of data directly to the cloud. Thanks for any suggestions.

FreshMike
  • 481
  • 1
  • 6
  • 26

2 Answers2

3

9 minutes is a hard limit for Cloud Functions that can't be exceeded. If you can't split up your work into smaller units, one for each function invocation, consider using a different product. Cloud Run limits to 15 minutes, and Compute Engine has no limit that would apply to you.

Doug Stevenson
  • 297,357
  • 32
  • 422
  • 441
-1

Google Cloud Scheduler may work well for that.

Here is a nice google blog post that shows example of how to set up a python script.

p.s. you would probably want to connect it to the App Engine for the actual execution.

Neil
  • 7,482
  • 6
  • 50
  • 56