1

I'm working with generics and encoders with Spark Datasets. And facing the above error with code that looks like. Please ignore the semantics of the code, Just posting a replicated simplified usecase.

The spark version I'm using is 3.2.1. And scala version 2.12.

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.node.ObjectNode
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.streaming.{StreamingQuery, Trigger}
import org.apache.spark.sql.{Dataset, Encoder, SparkSession}

abstract class SparkRunner[D](
                               deserializeFunc: Array[Byte] => Option[D])
                             (implicit spark: SparkSession, mapper: ObjectMapper)
  extends Serializable {
  import spark.implicits._

  val linesOfBytes: Dataset[Array[Byte]] = spark.readStream
    .format("socket")
    .option("host", "localhost")
    .option("port", 9999)
    .load()
    .as[String].map(mapper.writeValueAsBytes)


  implicit val encD:Encoder[D]
  implicit val encOD:Encoder[Option[D]]


  val transformedDf =
    linesOfBytes
      .map(deserializeFunc)

  def start(): StreamingQuery = {
    transformedDf.writeStream
      //todo: Adopt config builder pattern for sink and src properties eventually.
      .format("console")
      .queryName("Query1")
      .trigger(Trigger.ProcessingTime(0))
      .start()
  }

}

object SparkMain {

  def main(args: Array[String]): Unit = {

    implicit val spark: SparkSession = SparkSession
      .builder()
      .master("local")
      .appName("A")
      .getOrCreate()

    implicit val objMapper = new ObjectMapper()

    case class Payload(payload: ObjectNode)


    def deserialize(bytes: Array[Byte]): Option[String] = {
      objMapper.registerModule(DefaultScalaModule)
      val o = objMapper.readValue(bytes, classOf[Payload])

        Some(objMapper.writeValueAsString(o.payload))
    }

    try new SparkRunner(
      deserialize) {
//      override implicit val encD: Encoder[String] = org.apache.spark.sql.Encoders.kryo[String]
//      override implicit val encOD: Encoder[Option[String]] = org.apache.spark.sql.Encoders.kryo[Option[String]]
      override implicit val encD: Encoder[String] = ExpressionEncoder()
      override implicit val encOD: Encoder[Option[String]] = ExpressionEncoder()
    }.start().awaitTermination()
    finally {
      spark.close()
    }
  }
}

//SparkMain.main(Array("")) // Uncomment if you're running this in spark-shell

The whole of the above works if we replace generics with concrete types.

Hemanth Gowda
  • 604
  • 4
  • 16

1 Answers1

1
  lazy val transformedDf =
    linesOfBytes
      .map(deserializeFunc)

lazy (or def) is needed because your object isn't constructed yet, if you debug and step into the map call you'll see a null value for the implicit Encoder evidence field. As the Spark code is lazy you may as well use def.

In your code example you already pass in implicits, why not also pass those two (or four from the original SO post) into SparkRunner? They seem a perfect case for a type class.

That said you can also make the implicits both def in base class and impl and achieve the same result.

Chris
  • 1,240
  • 7
  • 8
  • Thanks for the answer and quick response. If I change the implicits to def, I see the below compile time error. Unable to find encoder for type Option[D]. An implicit Encoder[Option[D]] is needed to store Option[D] instances in a Dataset. Could you please elaborate or provide a link for what you mean by "They seem a perfect case for a type class". – Hemanth Gowda Jul 31 '23 at 16:46
  • 1
    sure see [this](https://scalac.io/blog/typeclasses-in-scala/) for an intro. You could have an interface with generic type D defined for all of your implicits (spark, mapper, the other four) which your class takes as a single implicit of type D, then import the implicits from that. You then have type class instances for any D which works with your specific encoders in them if you need other encoder types than kyro. – Chris Jul 31 '23 at 17:08