Creating Date Partitioned Tables programmatically

Question

Is there a way to create a date partitioned table using Apache Beam BigQueryIO, in other words, is there a way to use partition decorator for a table which is not created yet?

I know that I can create a table at first and then I can use partition decorator within my code but since I dynamically determine the TableDestination from fields of rows, I cannot create these tables in advance.

My code is like this:

rows.apply("Write rows",
    BigQueryIO.writeTableRows()
      .to(new SerializableFunction<ValueInSingleWindow<TableRow>, TableDestination>() {
          @Override
          public TableDestination apply(ValueInSingleWindow<TableRow> value) {
            TableRow t = value.getValue();

            String tableName = ... // get from the fields of table row
            String partition = ... // get the date part that will be used for decorator

            TableDestination td = new TableDestination(
                      "project-id:dataset-id." + tableName + "$" + partition, "");
              return td;
          }
      }).withSchema(someSchema)
      .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
    .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));

With this, it tries to create a table of project-id:dataset-id.tableName$partition and it complains that $ cannot be used inside a table name.

score 2 · Accepted Answer · answered Jun 08 '17 at 07:14

2

it seems that this currently is not possible.

There is a request for this in the official BEAM JIRA issue list though:BEAM-2390 and an official pull request , so it looks like this is going to be possible soon!

answered Jun 08 '17 at 07:14

Rogier Werschkull

684
5
15

score 0 · Answer 2 · answered Jun 06 '17 at 16:23

0

Table id format: The ID of the table. The ID must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_). The maximum length is 1,024 characters. check the BigQuery doc.

answered Jun 06 '17 at 16:23

rav

3,579
1
18
18

That `$` sign stands for partition decorator, which is the essence of my question. Normally, if I create a table beforehand, I can use that sign to specify my partition but I cannot use it for a table which is not created yet. – Ali Jun 07 '17 at 08:20

Creating Date Partitioned Tables programmatically

2 Answers2