Bigquery LoadJobConfig Delete Source Files After Transfer

Question

When creating a Bigquery Data Transfer Service Job Manually through the UI, I can select an option to delete source files after transfer. When I try to use the CLI or the Python Client to create on-demand Data Transfer Service Jobs, I do not see an option to delete the source files after transfer. Do you know if there is another way to do so? Right now, my Source URI is gs://<bucket_path>/*, so it's not trivial to delete the files myself.

Answer 1

For me works this snippet (replace YOUR-... with your data):

from google.cloud import bigquery_datatransfer
import os


os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "YOUR-CRED-FILE-PATH"


transfer_client = bigquery_datatransfer.DataTransferServiceClient()


destination_project_id = "YOUR-PROJECT-ID"
destination_dataset_id = "YOUR-DATASET-ID"


transfer_config = bigquery_datatransfer.TransferConfig(
    destination_dataset_id=destination_dataset_id,
    display_name="YOUR-TRANSFER-NAME",
    data_source_id="google_cloud_storage",
    params={
        "data_path_template":"gs://PATH-TO-YOUR-DATA/*.csv",
        "destination_table_name_template":"YOUR-TABLE-NAME",
        "file_format":"CSV",
        "skip_leading_rows":"1",
        "delete_source_files": True
    },
)


transfer_config = transfer_client.create_transfer_config(
    parent=transfer_client.common_project_path(destination_project_id),
    transfer_config=transfer_config,
)
print(f"Created transfer config: {transfer_config.name}")

In this example, table YOUR-TABLE-NAME must already exist in BigQuery, otherwise the transfer will crash with error "Not found: Table YOUR-TABLE-NAME".

I used this packages:

google-cloud-bigquery-datatransfer>=3.4.1
google-cloud-bigquery>=2.31.0

Pay attention to the attribute delete_source_files in params. From docs:

• Optional param delete_source_files will delete the source files after each successful transfer. (Delete jobs do not retry if the first effort to delete the source files fails.) The default value for the delete_source_files is false.