I just started on Bigquery projects, I was trying to connect Postgres SQL instance using Bigquery federated query. When I ran the query in Bigquery console (UI) it was running fine and the results came out:
SELECT * FROM EXTERNAL_QUERY("projects/project_XXX/locations/asia-southeast1/connections/connection_XXX", "SELECT * FROM postgres_table_name;")
But, when I put it on code using python airflow, it gave me an error:
404 Not found: Connection {connection_name}
Here's the code, my python scripts:
execute_query_save = BigQueryInsertJobOperator(
task_id="execute_query_insert",
configuration={
"query": {
"query": open(f"{SQL_PATH}/file_sql.sql", "r").read(),
"writeDisposition": "WRITE_TRUNCATE",
"useLegacySql": False,
"destinationTable": {
'projectId': "project_xxx",
'datasetId': "dataset_xxx",
'tableId': "table_target",
},
}
},
)
Any ideas on how I can work around this issue and what I missed ?