Stack overflow in typeclass with implicit conversion

Question

I made a generic DynamoFormat for Scanamo that would put any object that has Circe's Encoder and Decoder defined into database as a Json string.

import com.gu.scanamo.DynamoFormat
import io.circe.parser.parse
import io.circe.syntax._
import io.circe.{Decoder, Encoder}

object JsonDynamoFormat {    
  def forType[T: Encoder: Decoder]: DynamoFormat[T] = DynamoFormat.coercedXmap[T, String, Exception] {
    s => parse(s).flatMap(_.as[T]).fold(err => throw err, obj => obj)
  } {
    obj => obj.asJson.noSpaces
  }
}

Then I added an implicit conversion (to the same object JsonDynamoFormat) to automatically provide these formatters.

implicit def jsonToFormat[T: Encoder: Decoder]: DynamoFormat[T] = JsonDynamoFormat.forType[T]

When I import it, compiler resolves formatters successfully, however at runtime I get a stack overflow in JsonDynamoFormat, where calls to jsonToFormat and forType alternate infinitely:

Exception in thread "main" java.lang.StackOverflowError
    at JsonDynamoFormat$.forType(JsonDynamoFormat.scala:12)
    at JsonDynamoFormat$.jsonToFormat(JsonDynamoFormat.scala:9)
    at JsonDynamoFormat$.forType(JsonDynamoFormat.scala:13)
    at JsonDynamoFormat$.jsonToFormat(JsonDynamoFormat.scala:9)
    ...

I can't really understand what happens here. Could anyone shed a light on this?

I wonder what would happen if you make `def forType` implicit and remove `implicit def jsonToFormat`. Looks like the second one is pretty much redundant. — Haspemulator, Aug 25 '17 at 12:23
@Haspemulator, interestingly it fails to compile, although function signatures indeed seem the same. Weird. — Vasiliy Ivashin, Aug 25 '17 at 13:07
I think this failure is the key to get to root cause of the problem. Enable `scalacOptions ++= Seq("-Xlog-implicits")` to get the implicit resolution log (could be a lot of output), and optionally `libraryDependencies ++= Seq(compilerPlugin("io.tryp" %% "splain" % "0.2.4"))` to make that output nicer. That might help in explaining why an implicit not found. — Haspemulator, Aug 25 '17 at 13:16
It sounds like compiler decided to satisfy one function implicit parameter with another's value, hence circular dependency and stack overflow. You could try put these 2 functions in different scopes, so that only one sees another. — Haspemulator, Aug 25 '17 at 13:17
Making `forType` an `implicit def` failed to compile because of name conflict with a method imported from another class. After resolving name conflict it failed at runtime with infinite recursion :) This led me to search for other implicits, and I found that `coercedXmap` takes an implicit parameter of type `DynamoFormat[String]`, which of course resolved to `forType`. After providing a correct string formatter everything works as expected. Thanks again for your valuable suggestions regarding implicits debugging. If you wish to provide them as an answer, I'll be glad to accept it. — Vasiliy Ivashin, Aug 28 '17 at 10:01

score 2 · Accepted Answer · answered Aug 28 '17 at 11:39

Debugging Scala implicits errors can be quite taxing. Here is a couple of suggestions that can help:

Enable scalacOptions ++= Seq("-Xlog-implicits") compiler option. This will print implicit search log, and can be useful to understand where exactly the implicit chain breaks.
Add splain libraryDependencies ++= Seq(compilerPlugin("io.tryp" %% "splain" % "0.2.4")) to improve the implicit debug log readability.

In general, stack overflow at runtime with generically derived typeclasses is a sign of wrong implicit resolution. This usually means compiler has found a couple of circularly dependent implicits and used one of them to satisfy the other one, and vice versa.

Normally such situation is recognized at compile time, and compile produces "diverging implicits" error, but that error can be a false positive, and therefore library authors usually circumvent it by using a technique like Lazy typeclass from shapeless. However in case of an actual buggy circular implicits, this will result in runtime error, instead of compile time error.

Stack overflow in typeclass with implicit conversion

1 Answers1