1

we have one sql script and want to execute it in bigquery from apache airflow using BQ client. we are generating sql script using python function. while running this script if there is any issue then want to ignore that and proceed run only successful/valid statement. is it possible?

from google.cloud import bigquery
client = bigquery.Client()

script_t="""insert into table1 select col1 from db.tab2;
insert into table1 select col2 from db.tab3;"""

result=client.query(script_t)

now in this example suppose my db.tab2 is not present in db then whole script will get fail but i want to ignore error and move to next one and want to run insert into table1 select col2 from db.tab3;

even you have any work-around for this then please suggest :)

Yug
  • 105
  • 1
  • 9

1 Answers1

0

I would not recommend ignoring any issue. It might ignore the error you did not meant to ignore, like connection error or even syntax error.

What you can do instead is use BigQuery Scripting and INFORMATION_SCHEMA to make your script reliable even if table does not exist.

You can use information schema to check if table exists: https://cloud.google.com/bigquery/docs/information-schema-tables

And then use Scripting IF condition to only run INSERT command if it does: https://cloud.google.com/bigquery/docs/reference/standard-sql/scripting

Michael Entin
  • 7,189
  • 3
  • 21
  • 26