2

I am trying to create a custom Aggregator function producing a Map as the result, however it requires an Encoders. As referenced in https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Encoders.html, there isn't one for now.

Does anyone know a workaround for this? Thank you in advance!

Nguyễn Hải Hà
  • 31
  • 3
  • 1
    In scala you can use the handful `org.apache.spark.sql.catalyst.encoders.ExpressionEncoder()`, but it looks like it's a bit more complicated in java. You might find something useful [here](https://echauchot.blogspot.com/2020/03/how-to-create-custom-spark-encoder-in.html) – leleogere Aug 31 '22 at 09:42

1 Answers1

1

This is not exactly an answer, but if someone came here with the same question but in Scala instead of Java, they can use the ExpressionEncoder:

import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder

// ...

def outputEncoder: Encoder[Map[String, Int]] = ExpressionEncoder()

For the original question in Java, this link about how to write a custom encoder might help.

leleogere
  • 908
  • 2
  • 15