-1

I have a cloud functions that loads a dataframe to bigquery and want to allow field addtions

My job_config is as follow but I always get an error saying that the ALLOW FIELD ADDTION is only available with table partittions. The destination table is already partition by date so I dont really know how to get this work.

job_config = bigquery.LoadJobConfig(
      schema=[bigquery.SchemaField("fecha", bigquery.enums.SqlTypeNames.DATE)],
      write_disposition="WRITE_TRUNCATE"
      ,create_disposition = "CREATE_IF_NEEDED"
      ,time_partitioning = bigquery.table.TimePartitioning(field="fecha")
      ,schema_update_options = 'ALLOW_FIELD_ADDITION'
    )
Aleksei Díaz
  • 35
  • 5
  • Please specify your use case more clearly. – Aishwary Shukla Apr 01 '22 at 01:27
  • so I want my schema to be flexible so let say my bigquery table schema is made of 3 columns (col1, col2 and col3) but my dataframe has columns col1, col2 and col4. I want to be able to load that data and add to my bigquery table a new column named col4 – Aleksei Díaz Apr 02 '22 at 17:12
  • Try to access table partition directly in your config. Something like this: `myproject.dataset.table_20170101`. where 20170101 denotes the partition date. Your error specifically is saying to specify partition when using write_truncate. Also, remove schema and time_partitioning fields from your config and then run. Let me know if that works. – Aishwary Shukla Apr 04 '22 at 13:18
  • Hi @AlekseiDíaz, Did the above comment help you in resolving the issue? – Prajna Rai T Apr 06 '22 at 08:02
  • @AishwaryShukla if I pass the partition date in the table name, wouldn't it create a single table for that date? I will end up with multiple tables. Also, I have multiple dates within each load job so I don't really know how can I make it work. – Aleksei Díaz Apr 13 '22 at 15:44
  • @PrajnaRaiT sorry for the delay. I ended up finding out that there is no need to to specify schema update options. Since I set it as "Write_truncate" it will load the new schema adding as many columns as needed. – Aleksei Díaz Apr 13 '22 at 15:46

1 Answers1

0

Thre is no need to pass ,schema_update_options = 'ALLOW_FIELD_ADDITION' since the write_disposition="WRITE_TRUNCATE" will handle the schema changes.

Thank you all!

Aleksei Díaz
  • 35
  • 5