Implicit Encoder for TypedDataset and Type Bounds in Scala

Question

My objective is to create a MyDataFrame class that will know how to fetch data at a given path, but I want to provide type-safety. I'm having some trouble using a frameless.TypedDataset with type bounds on remote data. For example

sealed trait Schema
final case class TableA(id: String) extends Schema
final case class TableB(id: String) extends Schema

class MyDataFrame[T <: Schema](path: String, implicit val spark: SparkSession) {
  def read = TypedDataset.create(spark.read.parquet(path)).as[T]
}

But I keep getting could not find implicit value for evidence parameter of type frameless.TypedEncoder[org.apache.spark.sql.Row]. I know that TypedDataset.create needs an Injection for this to work. But I'm not sure how I would write this for a generic T. I thought the compiler would be able to deduce that since all subtypes of Schema are case classes that it would work.

Anybody ever run into this?

Dmytro Mitin · Accepted Answer · 2022-09-15T22:46:59.053

All implicit parameters should be in the last parameter list and this parameter list should be separate from non-implicit ones.

If you try to compile

class MyDataFrame[T <: Schema](path: String)(implicit spark: SparkSession) {
  def read = TypedDataset.create(spark.read.parquet(path)).as[T]
}

you'll see error

Error:(11, 35) could not find implicit value for evidence parameter of type frameless.TypedEncoder[org.apache.spark.sql.Row]
    def read = TypedDataset.create(spark.read.parquet(path)).as[T]

So let's just add corresponding implicit parameter

class MyDataFrame[T <: Schema](path: String)(implicit spark: SparkSession, te: TypedEncoder[Row]) {
  def read = TypedDataset.create(spark.read.parquet(path)).as[T]
}

we'll have error

Error:(11, 64) could not find implicit value for parameter as: frameless.ops.As[org.apache.spark.sql.Row,T]
    def read = TypedDataset.create(spark.read.parquet(path)).as[T]

So let's add one more implicit parameter

import frameless.ops.As
import frameless.{TypedDataset, TypedEncoder}
import org.apache.spark.sql.{Row, SparkSession}

class MyDataFrame[T <: Schema](path: String)(implicit spark: SparkSession, te: TypedEncoder[Row], as: As[Row, T]) {
  def read = TypedDataset.create(spark.read.parquet(path)).as[T]
}

or with kind-projector

class MyDataFrame[T <: Schema : As[Row, ?]](path: String)(implicit spark: SparkSession, te: TypedEncoder[Row]) {
  def read = TypedDataset.create(spark.read.parquet(path)).as[T]
}

You can create custom type class

  trait Helper[T] {
    implicit def te: TypedEncoder[Row]
    implicit def as: As[Row, T]
  }

  object Helper {
    implicit def mkHelper[T](implicit te0: TypedEncoder[Row], as0: As[Row, T]): Helper[T] = new Helper[T] {
      override implicit def te: TypedEncoder[Row] = te0
      override implicit def as: As[Row, T] = as0
    }
  }

  class MyDataFrame[T <: Schema : Helper](path: String)(implicit spark: SparkSession) {
    val h = implicitly[Helper[T]]
    import h._
    def read = TypedDataset.create(spark.read.parquet(path)).as[T]
  }

or

  class MyDataFrame[T <: Schema](path: String)(implicit spark: SparkSession, h: Helper[T]) {
    import h._
    def read = TypedDataset.create(spark.read.parquet(path)).as[T]
  }

or

  trait Helper[T] {
    def create(dataFrame: DataFrame): TypedDataset[T]
  }

  object Helper {
    implicit def mkHelper[T](implicit te: TypedEncoder[Row], as: As[Row, T]): Helper[T] =
      (dataFrame: DataFrame) => TypedDataset.create(dataFrame).as[T]
  }

  class MyDataFrame[T <: Schema : Helper](path: String)(implicit spark: SparkSession) {
    def read = implicitly[Helper[T]].create(spark.read.parquet(path))
  }

or

  class MyDataFrame[T <: Schema](path: String)(implicit spark: SparkSession, h: Helper[T]) {
    def read = h.create(spark.read.parquet(path))
  }

Corrected version:

import org.apache.spark.sql.Encoder
import frameless.{TypedDataset, TypedEncoder}

class MyDataFrame[T <: Schema](path: String)(implicit
  spark: SparkSession,
  e: Encoder[T],
  te: TypedEncoder[T]
) {
  def read: TypedDataset[T] = TypedDataset.create[T](spark.read.parquet(path).as[T])
}

or using context bounds

class MyDataFrame[T <: Schema : Encoder : TypedEncoder](path: String)(implicit
  spark: SparkSession
) {
  def read: TypedDataset[T] = TypedDataset.create[T](spark.read.parquet(path).as[T])
}

Testing: I converted a json file {"id": "xyz"} into parquet file and then

sealed trait Schema
final case class TableA(id: String) extends Schema
final case class TableB(id: String) extends Schema

import org.apache.spark.sql.SparkSession

implicit val spark: SparkSession = SparkSession.builder
  .master("local")
  .appName("Spark SQL basic example")
  .getOrCreate()

import spark.implicits._
import frameless.syntax._

val res: TypedDataset[TableA] = new MyDataFrame[TableA]("path/to/parquet/file").read
println(res) // [id: string]
res.foreach(println).run() // TableA(xyz)

Interesting, that works. Now I just need to understand what is going on. Thanks! — Daniel Severo, Apr 11 '19 at 16:46
Doesn't this force me to provide `te` and `as` for each instance I need to create? — Daniel Severo, Apr 11 '19 at 17:25
@DanielSevero I updated my answer. You can create custom type class. — Dmytro Mitin, Apr 11 '19 at 18:36
@DmytroMitin: I don't see how this could work this way. Reason: an evidence has to be provided at some point, be it a As[U,T] or Helper[T] or whatever. The "problem" is that, in this sort of implementation proposed by DanielSevero we don't have a concrete instance of Schema, since such type is meant for schema validation only. In other words, we need only a TypeTag which describes an instance of Schema, not an actual instance of Schema. Any ideas? Thanks a lot for your answer. Super useful. — Richard Gomes, Jun 14 '22 at 23:51
@RichardGomes Maybe you are right. Thanks. See the corrected version. — Dmytro Mitin, Sep 15 '22 at 22:41

Implicit Encoder for TypedDataset and Type Bounds in Scala

1 Answers1

Linked