7

I've created a Pipeline, which reads from a file in GCS, transforms it, and finally writes to a BQ table. The file contains a header row (fields).

Is there any way to programatically set the "number of header rows to skip" like you can do in BQ when loading in?

number of header rows to skip

Graham Polley
  • 14,393
  • 4
  • 44
  • 80

1 Answers1

4

This is not currently possible. It sounds like there are two potential requests here:

  • Specifying presence and skip behavior for header lines for a BigQuery import.
  • Specifying that a GCS text source should skip a header line.

Future work on this is tracked in https://issues.apache.org/jira/browse/BEAM-123.

Also, in the meantime, you could add a simple filter to your ParDo code to skip headers. Something like this:

PCollection<X> rows = ...;
PCollection<X> nonHeaders =
   rows.apply(Filter.by(new MatchIfNonHeader()));
Sam McVeety
  • 3,194
  • 1
  • 15
  • 38
  • Is there some kind of filter component that I can apply? Or do you just mean skipping the header in the actual "processElement" method of my ParDo code by checking if it's the header? – Graham Polley Feb 12 '15 at 01:55
  • one way could be start processing in PerDO and check if that line contains header because if does than skip it – Amit_Hora Jan 12 '17 at 06:02
  • was this ever addressed as a feature? – CCC Apr 29 '17 at 21:07