0

I am trying to create a pipeline which performs following task.

  1. read and parse the csv file
  2. apply schema on top of that
  3. records which are mapping schema is written to a valid bigquery table
  4. records which doesn't match schema (i.e. if column expect int but in file it's string) it goes to reject bucket.

I have write following pipeline. However, the problem is, I don't see any records going to either rejected or bigquery.

if schema is not matching, shouldn't it go to reject?

enter image description here

Gaurang Shah
  • 11,764
  • 9
  • 74
  • 137
  • Question, is the apply_schema step running, are the logs returning any information? – Eduardo Ortiz Feb 02 '22 at 23:07
  • Schema mapping does not automatically happen, but needs to be explicitly performed in the apply_schema step. In particular, it should be done using send-to-error directives to check if various fields are what you expect them to be. – Albert Shau Feb 08 '22 at 19:47

0 Answers0