2

In the following code I am trying to read a CSV file located in dataFile using Beam's TextIO and filter its header line, but I am getting a compile error with this message:

Error:(ROW, COLUMN) overloaded method value by with alternatives:
  [T, PredicateT <: org.apache.beam.sdk.transforms.SerializableFunction[T,Boolean]](x$1: PredicateT)org.apache.beam.sdk.transforms.Filter[T] <and>
  [T, PredicateT <: org.apache.beam.sdk.transforms.ProcessFunction[T,Boolean]](x$1: PredicateT)org.apache.beam.sdk.transforms.Filter[T]
 cannot be applied to (org.apache.beam.sdk.transforms.SimpleFunction[String,Boolean])
              .by(nonHeaderFilter))

Code:

val nonHeaderFilter: SimpleFunction[String, Boolean] = new SimpleFunction[String, Boolean]() {
    override def apply(input: String): Boolean = {
        input != MyClass.CsvHeader
    }
}

def readDataFile(input: PBegin, dataFile: String): PCollection[String] = {
    input
    .apply("Read Data File", TextIO.read().from(dataFile))
    .apply("Filter Header Line", Filter.by(nonHeaderFilter))
}

I think the problem is related to the fact that a SerializableFunction is a ProcessFunction and the SimpleFunction is a SerializableFunction. Somehow this is not handled correctly in scala.

Any recommendations to avoid this problem or did I misunderstand something?

Edit (workaround):

To resolve the issue temporarily, in case someone else faced this problem, I created a static Java method to provide the required filter:

import org.apache.beam.sdk.transforms.Filter;

public class BeamTransformProvider {

    public static Filter<String> notEqualFilter(String value) {
        return Filter.by(input -> !input.equals(value));
    }

}

which can be used as in the following:

def readDataFile(input: PBegin, dataFile: String): PCollection[String] = {
    input
    .apply("Read Data File", TextIO.read().from(dataFile))
    .apply("Filter Header Line", BeamTransformProvider.notEqualFilter(MyClass.CsvHeader))
}
Pablo
  • 10,425
  • 1
  • 44
  • 67
Mousa
  • 2,926
  • 1
  • 27
  • 35

0 Answers0