2

Does Google Datalab support time partitioned BigQuery table's specific partition as query result destination tables ? For example:

from gcp import bigquery as bq
queryString = 'SELECT "a1" AS col1'
tabNam = 'Feature.test$20141228'
bq.Query(queryString).execute(table_name=tabNam, table_mode='create')
shyam
  • 19
  • 3
  • Hi, Shyam, regarding creating partition tables from Datalab, what if the partition table doesn't exist yet, but I want to directly create a partition table using table.Create() ? do you know how to do it? thx. – Haipeng Su Jan 20 '17 at 16:09

2 Answers2

1

My guess is that this feature is not needed to be somehow specially supported by Datalab, because the only what you need to do is to supply table name with time partition as suffix (like you have it in your question - Feature.test$20141228). Of course you need to make sure first that your table (Feature.test) is properly configured with timePartitioning table's property

Mikhail Berlyant
  • 165,386
  • 8
  • 154
  • 230
  • 1
    Thank you @Mikhail ! @KS1 pointed out that instead of sending destination table name as a string, sending it as a dictionary works. Essentially, in the code snippet I mentioned above, replace: `tabNam = 'Feature.test$20141228'` by: `tabNam = {'table_id':'test$20141228', 'dataset_id':'Feature'}` I have verified, this approach works. – shyam Jul 18 '16 at 09:16
0

There is great feature for tables in Google Big Query. If you have several tables with the same name + (Date), e.g test20141228, test20141229... You will have a set of tables with a scroll down button, as shown in the pic, which is really nice. Then you can use wild card function TABLE_DATE_RANGE([Feature.test], date1, date2) to query tables between testdate1 ~ testdate2, which is also really nice.

enter image description here

Haipeng Su
  • 2,341
  • 2
  • 15
  • 30
  • Thank you @Haipeng Su ! We were forced to abandon `TABLE_DATE_RANGE()` and instead use time partitioned tables, as we hit the limit of 1000 tables in a single query which contained few joins. – shyam Jul 25 '16 at 07:24