using time partitioning for bigquery load doesn't upload every row

Question

I'm attempting to use the BigQuery python API client for uploading a large dataframe. The upload works however when specifying a time partition only some rows are uploaded. When time partitioning is omitted all rows are uploaded.

JOB_CONFIG:

job_config = bigquery.LoadJobConfig(write_disposition="WRITE_TRUNCATE",autodetect=True,)
job_config.time_partitioning = bigquery.TimePartitioning(type_=bigquery.TimePartitioningType.DAY, field="date")

dataframe is 120,532 rows, when table rows are checked in BigQuery only 62,433 rows are added to a table. I've checked the date column in the DataFrame to make sure every row has a date and I believe that is the case.

Any other ideas on what could be causing an incomplete upload?

df['date'].count() returns the correct 120,532 count of the size of the dataframe — 在去中国, May 31 '23 at 21:57

score 0 · Accepted Answer · answered Jun 05 '23 at 19:57

0

in BigQuery sandbox mode there is a 60 day partition expiration limit. I believe that when adding in partitioning the table is uploaded limited to 60 days without providing any message that it does so.

answered Jun 05 '23 at 19:57

在去中国

78
8

using time partitioning for bigquery load doesn't upload every row

1 Answers1