Scala macro compilation times

Question

Working on a scala project that uses libraries such as circe, enumeratum, and shapeless

We have a few enums defined namely, Region; EntityType and EntityAction eg

sealed trait EntityAction extends EnumEntry with Hyphencase

case object EntityAction extends Enum[EntityAction] with CirceEnum[EntityAction] {
  //noinspection ScalaStyle
  case object * extends EntityAction

  case object CreateUser extends EntityAction
  ... omitted ...
}

We also have a Permission model defined as

case class Permission(
                   resource: EntityType,
                   regions: Set[Region],
                   actions: Set[EntityAction],
                   instances: Set[String]
                 )

We also have an UserEntityModel (that is saved in our data store in a json format) and a UserModel that is returned as part of the http response

case class UserEntityV1(
                     override val entityKey: EntityKey,
                     firstName: String,
                     lastName: String,
                     override val email: String,
                     override val permissions: Set[Permission],
                     override val revokedPermissions: Set[Permission],
                     override val status: EntityStatus
                     ...

           ) extends UserEntity

EntityKey contains the Region enum and the omitted fields are of non enum types

case class User(
             override val uuid: Option[String] = None,
             firstName: String,
             lastName: String,
             email: String,
             status: EntityStatus,
             permissions: Set[Permission],
             revokedPermissions: Set[Permission]
             ...
           ) extends HasPayload

Without the permissions and revokedPermissions from the UserEntity and the User model, the clean compile time is under 3 minutes. With those fields included, the compiler runs into

java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.Arrays.copyOfRange(Arrays.java:3664)
    at java.lang.String.<init>(String.java:201)
    at java.lang.StringBuilder.toString(StringBuilder.java:407)
    at scala.tools.nsc.transform.Erasure.fullNameInSig$1(Erasure.scala:261)
    at scala.tools.nsc.transform.Erasure.classSig$1(Erasure.scala:292)
    at scala.tools.nsc.transform.Erasure.jsig$1(Erasure.scala:336)
    at scala.tools.nsc.transform.Erasure.boxedSig$1(Erasure.scala:243)
    at scala.tools.nsc.transform.Erasure.argSig$1(Erasure.scala:281)
    at scala.tools.nsc.transform.Erasure.$anonfun$javaSig$8(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.classSig$1(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.jsig$1(Erasure.scala:336)
    at scala.tools.nsc.transform.Erasure.boxedSig$1(Erasure.scala:243)
    at scala.tools.nsc.transform.Erasure.argSig$1(Erasure.scala:281)
    at scala.tools.nsc.transform.Erasure.$anonfun$javaSig$8(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.classSig$1(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.jsig$1(Erasure.scala:336)
    at scala.tools.nsc.transform.Erasure.boxedSig$1(Erasure.scala:243)
    at scala.tools.nsc.transform.Erasure.argSig$1(Erasure.scala:281)
    at scala.tools.nsc.transform.Erasure.$anonfun$javaSig$8(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.classSig$1(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.jsig$1(Erasure.scala:336)
    at scala.tools.nsc.transform.Erasure.boxedSig$1(Erasure.scala:243)
    at scala.tools.nsc.transform.Erasure.argSig$1(Erasure.scala:281)
    at scala.tools.nsc.transform.Erasure.$anonfun$javaSig$8(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.classSig$1(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.jsig$1(Erasure.scala:336)
    at scala.tools.nsc.transform.Erasure.boxedSig$1(Erasure.scala:243)
    at scala.tools.nsc.transform.Erasure.argSig$1(Erasure.scala:281)
    at scala.tools.nsc.transform.Erasure.$anonfun$javaSig$8(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.classSig$1(Erasure.scala:295)
    at scala.tools.nsc.transform.Erasure.jsig$1(Erasure.scala:336)
    at scala.tools.nsc.transform.Erasure.jsig$1(Erasure.scala:359)

We solved it by defining a .sbtopts played around with a few variations of it , as it stands the current .sbtopts is

-J-Xms1024M
-J-Xmx4G
-J-XX:ReservedCodeCacheSize=256M
-J-XX:MaxMetaspaceSize=1024M
-J-Xss2M

played around with diff Xss values as well as specifying -mem , does not seem to make much of a diff.

However, with those fields included, the clean compile time average is now almost 7 minutes (as the user entity is used in a few places)

All indications point to macro compilation. We are using auto derivation. Is there a way to keep things boilerplate free and keep the type safety of the enums but reduce the compile time.

Current versions are

scala version =  2.12.1
sbt version = 0.13.13 
circeVersion = "0.7.0"
enumeratumVersion = "1.5.10"
enumeratumCirceVersion = "1.5.10"
shapelessVersion = "2.3.2"

On updating to the latest versions of the library and scala 2.12.3, the compile time jumps to almost 20 minutes

Tried following some steps from debugging slow compile time

As far as I can keep the its the typer phase

@StephenCarman I have updated the original post with the .sbtopts. Played around with a few different options, I know this following example is java `https://github.com/lagom/activator-lagom-java` but i used that as a starting point — Omar A, Aug 16 '17 at 04:15
I meant that I used the .sbtopts from that as a starting point, not for the whole project, also had a look at `https://medium.com/@jan______/sbtconfig-is-deprecated-650d6ff10236` and `https://github.com/miguno/kafka-storm-starter/blob/develop/.sbtopts`. I did have `-J-XX:+UseConcMarkSweepGC -J-XX:+CMSClassUnloadingEnabled` in my .sbtopts file at some point as well — Omar A, Aug 16 '17 at 04:45
Have you tried annotating classes you need JSON-ified with `@JsonCodec` ? — Oleg Pyzhcov, Aug 16 '17 at 12:50
I changed the full auto derviation from using `import io.circe.generic.auto._` to use `@JsonCodec` instead. It has sped up the compile time by a bit. Still need to do a few more iterations to get an average. Had to change a few things in the code to make it compile. It might have pointed to the possible culprit. I am using circes `withDiscriminator("typ")` from the circes extras. and using @jsonCodec breaks the discriminator column which is no longer being written out to my json. And I have to specify the actual entity class instead of the sealed trait that it inherits from in a few places. — Omar A, Aug 17 '17 at 05:16
@OlegPyzhcov so I had to use the `@ConfiguredJsonCodec` annotation to fix the discriminator being broken but looks like going away from full auto derivation has fixed the compilation time issues. `clean compile` seems to be under a minute now. Thanks!! — Omar A, Aug 20 '17 at 23:50

Scala macro compilation times

0 Answers0