3

Currently using GCP BigQuery UI for scheduled queries, everything is manually done.

Wondering if there's a way to automatically deploy to GCP using a config JSON that contains the scheduled query's parameters and scheduled times through github actions?

So far, this is one option I've found that makes it more "automated":

- store query in a file on Cloud Storage. When invoking Cloud Function, you read the file content and you perform a bigQuery job on it.
    - have to update the file content to update the query
    - con: read file from storage, then call BQ: 2 api calls and query file to manage

Currently using DBT in other repos to automate and make this process quicker: https://docs.getdbt.com/docs/introduction

Would prefer the github actions version though, just haven't found a good docu yet :)

iNano
  • 58
  • 7
  • Terraform doesn't handle scheduled queries in GCP so my suspicion is it can't be done at the moment, but I'll be following this with the hope that someone proves my suspicions wrong. – Sylvansight Sep 28 '21 at 07:15
  • 1
    Terraform had a request in 2018 regarding this and it looksl ike it was been updated. Never tried, but here's the doc:(https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/bigquery_data_transfer_config#example-usage---bigquerydatatransfer-config-scheduled-query) – iNano Sep 29 '21 at 11:27
  • 1
    @iNano just curious - why do you prefer to use Github Actions for this instead of the current dbt approach? – Jakub Kukul Nov 10 '21 at 14:47
  • @iNano were you able to deploy it using github actions. – Urvah Shabbir Jan 28 '22 at 07:11

0 Answers0