Trouble with type variance

Question

So, let's say I have a class with a contravariant type parameter:

  trait Storage[-T] {
    def list(path: String): Seq[String]
    def getObject[R <: T](path: String): Future[R]
  }

The idea of the type parameter is to constrain the implementation to the upper boundary of types that it can return. So, Storage[Any] can read anything, while Storage[avro.SpecificRecord] can read avro records, but not other classes:

def storage: Storage[avro.SpecificRecord] 
storage.getObject[MyAvroDoc]("foo") // works
storage.getObject[String]("bar") // fails

Now, I have a utility class that can be used to iterate through objects in a given location:

class StorageIterator[+T](
  val storage: Storage[_ >: T], 
  location: String
)(filter: String => Boolean) extends AbstractIterator[Future[T]] {
  val it = storage.list(location).filter(filter)
  def hasNext = it.hasNext
  def next = storage.getObject[T](it.next)
}

This works, but sometimes I need to access the underlying storage from the iterator downstream, to read another type of object from an aux location:

def storage: Storage[avro.SpecificRecord]
val iter = new StorageIterator[MyAvroDoc]("foo")
iter.storage.getObject[AuxAvroDoc](aux)

This does not work, of course, because storage type parameter is a wildcard, and there is no proof that it can be used to read AuxAvroDoc

I try to fix it like this:

class StorageIterator2[P, +T <: P](storage: Storage[P])
  extends StorageIterator[T](storage)

This works, but now I have to specify two type params when creating it, and that sucks :( I tried to work around it by adding a method to the Storage itself:

trait Storage[-T] {
  ... 
  def iterate[R <: T](path: String) = 
    new StorageIterator2[T, R](this, path)
}

But this doesn't compile because it puts T into an invariant position :( And if I make P contravariant, then StorageIterator2[-P, +T <: P] fails, because it thinks that P occurs in covariant position in type P of value T.

This last error I don't understand. Why exactly cannot P be contravariant here? If this position is really covariant (why is it?) then why does it allow me to specify an invariant parameter there?

Finally, does anyone have an idea how I can work around this? Basically, the idea is to be able to

1. Do storage.iterate[MyAvroDoc] without having to give it the upper boundary again, and
2. Do iterator.storage.getObject[AnotherAvroDoc] without having to cast the storage to prove that it can read this type of object.

Any ideas are appreciated.

Answer 1

StorageIterator2[-P, +T <: P] fails because it is nonsensical. If you have a StorageIterator2[Foo, Bar], and Bar <: Foo, then because it is contravariant in the first parameter, it is also a StorageIterator[Nothing, Bar], but Nothing has no subtypes, so it is logically impossible that Bar <: Nothing, yet this is what must be true to have a StorageIterator2. Therefore, StorageIterator2 cannot exist.

The root problem is that Storage should not be contravariant. Think of what a Storage[T] is, in terms of the contract it gives to its users. A Storage[T] is an object that you give paths to, and will output Ts. It makes perfect sense for Storage to be covariant: something that knows how to output Strings, for example, is also outputting Anys, so it makes logical sense that Storage[String] <: Storage[Any]. You say that it should be the other way around, that a Storage[T] should know how to output any subtype of T, but how would that work? What if someone adds a subtype to T after the fact? T can even be final and still have this problem, because of singleton types. That is unnecessarily complicated, and is reflected in your problem. That is, Storage should be

trait Storage[+T] {
  def list(path: String]: Seq[String]
  def get(path: String): T
}

This does not open you up to the example mistake you gave in your question:

val x: Storage[avro.SpecificRecord] = ???
x.get(???): avro.SpecificRecord // ok
x.get(???): String // nope
(x: Storage[String]).get(???) // nope

Now, your issue is that you can't do something like storage.getObject[T] and have the cast be implicit. You can instead a match:

storage.getObject(path) match {
  case value: CorrectType => ...
  case _ => // Oops, something else. Error?
}

a plain asInstanceOf (undesirable), or you can add a helper method to Storage, like the one you had before:

def getCast[U <: T](path: String)(implicit tag: ClassTag[U]): Option[U] = tag.unapply(get(path))