1

I want to schedule my dataflow sql job. this is my gcloud command

gcloud dataflow sql query 'My SELECT query' --job-name dfsql-2d7a0189-177fc4f9444 --region us-east4 --bigquery-write-disposition write-truncate --bigquery-project dev-dp-myproject --bigquery-dataset mydataset --bigquery-table mytable --service-account-email dataflow-run @myserviceaccount.iam.gserviceaccount.com --subnetwork https://www.googleapis.com/compute/v1/projects/mysubnetwork

how to schedule this gcloud commmand so that every 15 minutes it runs.

user3796386
  • 21
  • 2

2 Answers2

1

An alternative solution, you can run your query through Apache beam Dataflow runner. As Dataflow jobs can be easily be scheduled through Cron/Google Scheduler/Google Composer as per your requirement. A sample Apache beam code, you can refer it and customize it accordingly.

Vibhor Gupta
  • 670
  • 7
  • 16
0

Cloud Shell isn't a tool intended to be used as part of your production environment, the recommended in this types of cases is that you use the tools provided by the GCP products themselves instead of running a gcloud command.

You can find in this tutotial an example of how to schedule Dataflow Jobs with Cloud Scheduler, which I believe is that you are looking for.

Ralemos
  • 5,571
  • 2
  • 9
  • 18
  • thank you for you response. The link which you shared is useful for Dataflow jobs written using JAVA sdk. I am using DATAFLOW SQL to create the job, and not sure how to create template for this job, for the scheduling – user3796386 Mar 06 '21 at 15:23
  • I think the solution proposed by Vibhor above would be a good option in that case. – Ralemos Mar 09 '21 at 16:21