In the following code I am trying to read a CSV file located in dataFile
using Beam's TextIO
and filter its header line, but I am getting a compile error with this message:
Error:(ROW, COLUMN) overloaded method value by with alternatives:
[T, PredicateT <: org.apache.beam.sdk.transforms.SerializableFunction[T,Boolean]](x$1: PredicateT)org.apache.beam.sdk.transforms.Filter[T] <and>
[T, PredicateT <: org.apache.beam.sdk.transforms.ProcessFunction[T,Boolean]](x$1: PredicateT)org.apache.beam.sdk.transforms.Filter[T]
cannot be applied to (org.apache.beam.sdk.transforms.SimpleFunction[String,Boolean])
.by(nonHeaderFilter))
Code:
val nonHeaderFilter: SimpleFunction[String, Boolean] = new SimpleFunction[String, Boolean]() {
override def apply(input: String): Boolean = {
input != MyClass.CsvHeader
}
}
def readDataFile(input: PBegin, dataFile: String): PCollection[String] = {
input
.apply("Read Data File", TextIO.read().from(dataFile))
.apply("Filter Header Line", Filter.by(nonHeaderFilter))
}
I think the problem is related to the fact that a SerializableFunction
is a ProcessFunction
and the SimpleFunction
is a SerializableFunction
. Somehow this is not handled correctly in scala.
Any recommendations to avoid this problem or did I misunderstand something?
Edit (workaround):
To resolve the issue temporarily, in case someone else faced this problem, I created a static Java method to provide the required filter:
import org.apache.beam.sdk.transforms.Filter;
public class BeamTransformProvider {
public static Filter<String> notEqualFilter(String value) {
return Filter.by(input -> !input.equals(value));
}
}
which can be used as in the following:
def readDataFile(input: PBegin, dataFile: String): PCollection[String] = {
input
.apply("Read Data File", TextIO.read().from(dataFile))
.apply("Filter Header Line", BeamTransformProvider.notEqualFilter(MyClass.CsvHeader))
}