4

I am trying to setup a schedule job/process in cloud to load csv data into Bigquery from google buckets using a python script. I have manage to get hold off the python code to do this but not sure where do I need to save this code so that this task could be completed as an automated process rather than running the gsutil commands manualy.

ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
LondonUK
  • 437
  • 1
  • 8
  • 20
  • What "cloud" are you trying to run your code in? Google Bigquery is only capable of holding data and running queries against it. It's not designed to run arbitrary code, there's regular Google Cloud for that. – ivan_pozdeev Oct 13 '16 at 09:34
  • Hi Ivan thanks for quick reply, We are using google cloud platform. I am aware that the bigquery is only capable of holding data and it is not possible to configure any other code script into it. I just want to where should I save my python script file on google cloud platform and create a job to execute this script and how to configure this process. – LondonUK Oct 13 '16 at 09:39
  • this doesn't work for you? https://cloud.google.com/appengine/docs/python/config/cron – HassenPy Oct 13 '16 at 09:57

1 Answers1

3

Reliable Task Scheduling on Google Compute Engine  |  Solutions  |  Google Cloud Platform, the 1st link in Google on "google cloud schedule a cron job", gives a high-level overview. Scheduling Tasks With Cron for Python  |  App Engine standard environment for Python  |  Google Cloud Platform, the 2nd link, has step-by-step instructions. They boil down to:

  • Create cron.yaml in the specificed format alongside your app.yaml
  • optionally test it at a development server
  • upload it to the Google Cloud with appcfg.py update or update_cron
ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
  • 1
    I found that this solution works on top of the app setting. Since I just created a script and I'm not very familiar with app development, I'm still working on how to setup that script on that app setting... or how to create the app from which I can call the scripts. Does anyone have a reference on how to do that? Thanks. – Diego-MX Feb 22 '19 at 23:18