1

I just started on Bigquery projects, I was trying to connect Postgres SQL instance using Bigquery federated query. When I ran the query in Bigquery console (UI) it was running fine and the results came out:

SELECT * FROM EXTERNAL_QUERY("projects/project_XXX/locations/asia-southeast1/connections/connection_XXX", "SELECT * FROM postgres_table_name;")

But, when I put it on code using python airflow, it gave me an error:

404 Not found: Connection {connection_name}

Here's the code, my python scripts:

execute_query_save = BigQueryInsertJobOperator(
    task_id="execute_query_insert",
    configuration={
        "query": {
            "query": open(f"{SQL_PATH}/file_sql.sql", "r").read(),
            "writeDisposition": "WRITE_TRUNCATE",
            "useLegacySql": False,
            "destinationTable": {
                'projectId': "project_xxx",
                'datasetId': "dataset_xxx",
                'tableId': "table_target",
            },
        }
    },
)

Any ideas on how I can work around this issue and what I missed ?

Amit Dash
  • 584
  • 8
  • 21

0 Answers0