BigQueryIO.writeTableRows writes to BigQuery with very high delay

Question

The following code snippet shows the writing method to BigQuery (it picks up data from PubSub). The "Write to BigQuery" dataflow step receives the TableRow data but it writes to BigQuery with very high delay (more than 3-4 hours) or doesn't even write the data at all. There are no errors/warnings in log and I can see the data arrived here. I wanted it as generic as possible so don't want to provide the schema in the code. Does this code snippet have any bug that can cause this behavior?

PCollection<TableRow> tableRows;
...
tableRows.apply("Write to BigQuery",
      BigQueryIO.writeTableRows().to(options.getTable())
      .withExtendedErrorInfo()
      .withCreateDisposition(CreateDisposition.CREATE_NEVER)
      .withWriteDisposition(WriteDisposition.WRITE_APPEND)
      .withMethod(Method.STREAMING_INSERTS)
      .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors()));

UPDATE: I modified the code to this:

tableRows.apply("Write to BigQuery",
  BigQueryIO.writeTableRows().to(options.getTable())
  .withCreateDisposition(CreateDisposition.CREATE_NEVER));

And now it works correctly. What was the problem with the original version of code?

Are you sure the data isn't there, it might be in the streaming buffer? You can check this by doing 'SELECT * FROM table WHERE _PARTITIONTIME IS NULL' — Richard Deurwaarder, Jun 06 '20 at 18:06
Can you provide more details on how you checked for the delay in Option 1 vs Option 2. — Jayadeep Jayaraman, Jun 07 '20 at 06:11

score 0 · Answer 1 · answered Jun 07 '20 at 19:26

0

Problem has been solved, it was infra problem.

answered Jun 07 '20 at 19:26

user2695543

81
4
22

BigQueryIO.writeTableRows writes to BigQuery with very high delay

1 Answers1