Generic type parameters for conversion to parallel collection

Question

I have this function to convert an Array to a ParArray, giving the number of threads as parameter:

def parN[T](collection: Array[T], n: Int) = {
    val parCollection = collection.par
    parCollection.tasksupport = new ForkJoinTaskSupport(
      new concurrent.forkjoin.ForkJoinPool(n))
    parCollection
}

Now I'd like to make this generic, such that it works with collections other than Array:

def parN[S, T[S] <: Parallelizable[S, ParIterable[S]]](collection: T[S], n: Int) = {
    val parCollection = collection.par
    parCollection.tasksupport = new ForkJoinTaskSupport(
      new concurrent.forkjoin.ForkJoinPool(n))
    parCollection
}

But when I call it with parN(Array(1, 2, 3), 2), I get this error:

inferred type arguments [Int,Array] do not
conform to method parN's type parameter bounds
[S,T[S] <: scala.collection.Parallelizable[S,scala.collection.parallel.ParIterable[S]]]

On the other hand, this is working:

val x: Parallelizable[Int, ParIterable[Int]] = Array(1, 2, 3)

Any ideas what might be wrong with my type parameters?

gourlaysama · Accepted Answer · 2013-05-24T21:00:52.983

First of all, note that this is a problem specific to Array: your method works for List or any other normal collection (except those with multiple type parameters, or none). Example:

scala> parN(List(1,2,3), 2)
res17: scala.collection.parallel.ParIterable[Int] = ParVector(1, 2, 3)

scala> parN(Set(1,2,3), 2)
res18: scala.collection.parallel.ParIterable[Int] = ParSet(1, 2, 3)

Array is always a special case when it comes to collections, because... it is not a collection. Because of Java, its definition is final class Array[T] extends Serializable with Cloneable. However, there exist two implicits, available everywhere, that can convert Array to a collection type (ArrayOps and WrappedArray). And these types do implement Parallelizable, so everything should be fine... except type inference gets in the way:

Your type parameter is only defined as T[S], so when receiving an Array[Int], it will gladly infer Array[Int], and then check the bounds: failure, Array doesn't extend Parallelizable. Game over.

I see two options:

You can explicitly say that you want a Parallelizable:

def parN[S](collection: Parallelizable[S, ParIterable[S]], n: Int)

Or if you need access to the actual type T (in your case no, but who knows):

def parN[S, T[S] <: Parallelizable[S, ParIterable[S]]](collection: T[S] with
   Parallelizable[S, ParIterable[S]], n: Int)

or you can accept anything that can be implicitly converted to a Parallelizable, using an implicit parameter:

def parN[S, T <% Parallelizable[S, ParIterable[S]]](collection: T, n: Int)

which is the short version of:

def parN[S, T](collection: T, n: Int)(implicit ev: T => 
  Parallelizable[S, ParIterable[S]])

All of those should work. In your case I would recommend the very first one: it is the most readable one that does the job.

WOw, beat me to it. The type-class pattern is the way to go here. However, don't you need to add in a `: Manifest` requirement to work with Arrays? — wheaties, May 24 '13 at 13:09
@wheaties no a `Manifest\ClassTag` isn't even needed because his method isn't actually creating an array (`Parallelizable.par` doesn't need a manifest, it always knows how to build a par collection). And in the case of arrays, it only wraps the original array, it doesn't even copy it. — gourlaysama, May 24 '13 at 13:16
Thanks for the detailed answer, I ended up using a `<%` view bound in combination with `CustomParallelizable`. — ValarDohaeris, May 24 '13 at 13:56

Generic type parameters for conversion to parallel collection

1 Answers1