How to work with generics of nested types in scala

Question

I'm building a small API around some spark code and I am finding myself creating traits like the following:

trait DataSource[T, F] {
    def run(config: DataSourceConfig[T,F]): DataSourceResource[F]
}

Take type T to represent RDD[F]. Do you see the problem here? It's getting obnoxious here having to force users to define both types like this:

class MySource extends DataSource[RDD[String], String] {


}

I understand scala has some black magic when it comes to generics that may allow me to specify that F is really a type nested within T but I can't seem to figure out how it would be done.

Thing is- T could also be a DStream[String]. RDD and DStream do not inherit from the same trait or base class.

Any ideas? It would really clean up my API if I could just specify that F is nested within T and still be able to use F everywhere.

score 1 · Answer 1 · edited May 23 '17 at 12:28

Well, this seems to work:

import scala.language.higherKinds

trait DataSource[A[_], B] {
  def foo(x: A[B]): Unit
}

class MySource extends DataSource[List, String] {
  def foo(x: List[String]) = println(s"$x")
}

But I look forward to seeing if someone else can point out how to restrict DataSource further so that only A[B] are allowed (somehow!).

EDIT: There's a discussion of scala's higher-kinded-type inference here: What are the limitations on inference of higher-kinded types in Scala?

How to work with generics of nested types in scala

1 Answers1