4

When creating a partitioned table using bq mk --time_partitioning_type=DAY are the partitions created based on the load time of the data, not a date key within the table data itself?

To create partitions based on dates within the date, is the current approach to manually create sharded tables, and load them based on date, as in this post from 2012?

Community
  • 1
  • 1
kermatt
  • 1,585
  • 2
  • 16
  • 36

2 Answers2

3

Yes, partitions created based on data load time not based on data itself
You can use partition decorator (mydataset.mytable1$20160810) if you want to load data into specific partition

Per my understanding, partition by column is something that we should expect to be supported at some point - but not now

Mikhail Berlyant
  • 165,386
  • 8
  • 154
  • 230
  • Adding a clarification -- the post from 2012 discusses creating sharded tables. You can load/append/copy to a specific partition of the table. So you'd have a single table but with multiple dated partitions: https://cloud.google.com/bigquery/docs/creating-partitioned-tables#restating_data_in_a_partition – Pavan Edara Aug 10 '16 at 17:27
  • To confirm, I can create and populate new paritions simply using a table$YYYYMMDD destination based on my date column values? – kermatt Aug 10 '16 at 17:37
3

Good news, BigQuery currently supports 2 type data partition, included partition by column. Please check here.

I like the feature: An individual operation can commit data into up to 2,000 distinct partitions.

Timothy Liu
  • 31
  • 4