Scio saveAsTypedBigQuery write to a partition for SCollection of Typed Big Query case class

Question

I'm trying to write a SCollection to a partition in Big Query using:

import java.time.LocalDate
import java.time.format.DateTimeFormatter
val date = LocateDate.parse("2017-06-21")
val col = sCollection.typedBigQuery[Blah](query)

col.saveAsTypedBigQuery(
      tableSpec = "test.test$" + date.format(DateTimeFormatter.ISO_LOCAL_DATE),
      writeDisposition = WriteDisposition.WRITE_EMPTY,
      createDisposition = CreateDisposition.CREATE_IF_NEEDED)

The error I get is Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."

How can I write to a partition? I don't see any options to specify partitions via either saveAsTypedBigQuery method so I was trying the Legacy SQL table decorators.

score 3 · Accepted Answer · answered Jun 18 '18 at 21:02

See: BigqueryIO Unable to Write to Date-Partitioned Table. You need to manually create the table. BQ IO cannot create a table and partition it.

Additionally, the no table decorators was a complete ruse. It's the alphanumeric part I was missing.

col.saveAsTypedBigQuery(
      tableSpec = "test.test$" + date.format(DateTimeFormatter.BASIC_ISO_DATE),
      writeDisposition = WriteDisposition.WRITE_APPEND,
      createDisposition = CreateDisposition.CREATE_NEVER)

Scio saveAsTypedBigQuery write to a partition for SCollection of Typed Big Query case class

1 Answers1