Using -= with incompatible types (Double and DenseVector) causes strange scala compiler error

Question

I am rather new to scala and having encountered an initially completely uninformative NullPointerException by the scala compiler I was now able to narrow down the issue and remove it's cause, but I still don't fully understand the compiler's behavior.

Environment:

IntelliJ IDEA 2020.2.3, JDK 1.8.0_271, Scala 2.13.3, SBT 1.3.13, Breeze 1.1

Edit: Verified without IntelliJ using sbt compile from the command line with SBT 1.4.2, compiler behavior is exactly identical

What I am currently observing:

MWE:

import breeze.linalg._
object MWE {
  def test(v: DenseVector[Double], m: DenseMatrix[Double]): Unit = {
    var value = 0.5
    value += v
    value += 0.5 * v.t * m * v
 }
}

Incorrectly using the operator += on the Double value and the DenseVector v in the line

    value += v

promptly and properly highlights this in red in the IDE. Trying to compile it, I get the useful info

value += is not a member of Double
  Expression does not convert to assignment because:
    overloaded method + with alternatives:
      (x: Double)Double <and>
      (x: Float)Double <and>
      (x: Long)Double <and>
      (x: Int)Double <and>
      (x: Char)Double <and>
      (x: Short)Double <and>
      (x: Byte)Double
     cannot be applied to (breeze.linalg.DenseVector[Double])
    expansion: value = value.<$plus: error>(v)
    value += v

Incorrectly using the operator -= instead, the IDE doesn't highlight it in red. Trying to compile it, I get a different but still useful info:

value -= is not a member of Double
  Expression does not convert to assignment because:
    type mismatch;
     found   : breeze.linalg.DenseVector[Double]
     required: Double
    expansion: value = value.<$minus: error>(v)
    value -= v

First question: Why do these two cases show different behavior? I had a short look at the handling of these operators in scala (see e.g. Where is Scala's += defined in the context of Int?) and after looking at the scala documentation (especially https://www.scala-lang.org/api/2.13.3/scala/Double.html) I don't see any reason for += and -= to be treated differently?

Going a step back towards our original code, the difference in behavior manifests in another way:

Using += in the line

value += 0.5 * v.t * m * v

results in similar behavior to the more simple case described above, as the operations on the right side produce a result of type DenseVector (more on this further below) and we try to add it to a Double. The operator is marked in red in the IDE and the same compiler error is shown.

Using -= however shows a completely different compiler error than in the more simple case:

scalac: Error while emitting MWE.scala
assertion failed: 
  Bad superClass for class Double: <none>
     while compiling: C:\Users\bt306644\typeBugMWE\src\main\scala\MWE.scala
        during phase: jvm
     library version: version 2.13.3
    compiler version: version 2.13.3
  reconstructed args: -classpath C:\Program Files\Java\jdk1.8.0_271\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_271\jre\lib\rt.jar;C:\Users\bt306644\typeBugMWE\target\scala-2.13\classes;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\com\chuusai\shapeless_2.13\2.3.3\shapeless_2.13-2.3.3.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\com\github\fommil\netlib\core\1.1.2\core-1.1.2.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\com\github\wendykierp\JTransforms\3.1\JTransforms-3.1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\net\sf\opencsv\opencsv\2.3\opencsv-2.3.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\net\sourceforge\f2j\arpack_combined_all\0.1\arpack_combined_all-0.1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\commons\commons-math3\3.5\commons-math3-3.5.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scala-lang\modules\scala-collection-compat_2.13\2.1.1\scala-collection-compat_2.13-2.1.1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scala-lang\scala-library\2.13.3\scala-library-2.13.3.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scala-lang\scala-reflect\2.13.3\scala-reflect-2.13.3.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scalanlp\breeze-macros_2.13\1.1\breeze-macros_2.13-1.1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scalanlp\breeze_2.13\1.1\breeze_2.13-1.1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\slf4j\slf4j-api\1.7.5\slf4j-api-1.7.5.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\typelevel\algebra_2.13\2.0.0-M2\algebra_2.13-2.0.0-M2.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\typelevel\cats-kernel_2.13\2.0.0-M4\cats-kernel_2.13-2.0.0-M4.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\typelevel\machinist_2.13\0.6.8\machinist_2.13-0.6.8.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\typelevel\spire-macros_2.13\0.17.0-M1\spire-macros_2.13-0.17.0-M1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\typelevel\spire-platform_2.13\0.17.0-M1\spire-platform_2.13-0.17.0-M1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\typelevel\spire-util_2.13\0.17.0-M1\spire-util_2.13-0.17.0-M1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\typelevel\spire_2.13\0.17.0-M1\spire_2.13-0.17.0-M1.jar;C:\Users\bt306644\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\pl\edu\icm\JLargeArrays\1.5\JLargeArrays-1.5.jar

  last tree to typer: Literal(Constant(0.5))
       tree position: line 6 of C:\Users\bt306644\typeBugMWE\src\main\scala\MWE.scala
            tree tpe: Double(0.5)
              symbol: null
           call site: constructor MWE in object MWE in package <empty>

== Source file context for tree position ==

     3   def test(v: DenseVector[Double], m: DenseMatrix[Double]): Unit = {
     4     var value = 0.5
     5 //    value += v
     6     value -= 0.5 * v.t * m * v
     7  }
     8 }
     9

Second question: What exactly is that supposed to tell me and why does it occur?

Besides these main questions some more context information and side notes:

Another step back towards the original code and setup in which I encountered this issue, we were actually using Scala 2.11.11 instead of 2.13.3 (and therefore Breeze 0.13.2 instead of Breeze 1.1). This caused the compiler to not display any useful information at all at the original second case, but instead produced a NullPointerException in the compiler:

scalac: Error: org.jetbrains.jps.incremental.scala.remote.ServerException
java.lang.NullPointerException
    at scala.tools.nsc.transform.Mixin$MixinTransformer.scala$tools$nsc$transform$Mixin$MixinTransformer$$postTransform(Mixin.scala:1154)
    at scala.tools.nsc.transform.Mixin$MixinTransformer$$anonfun$transform$1.apply(Mixin.scala:1261)
    at scala.tools.nsc.transform.Mixin$MixinTransformer$$anonfun$transform$1.apply(Mixin.scala:1261)
    at scala.reflect.internal.SymbolTable.enteringPhase(SymbolTable.scala:235)
    at scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1261)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1347)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1258)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1386)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1258)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2589)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2587)
    at scala.collection.immutable.List.loop$1(List.scala:176)
    at scala.collection.immutable.List.mapConserve(List.scala:200)
    at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2587)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1366)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1258)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.reflect.internal.Trees$$anonfun$itransform$2.apply(Trees.scala:1363)
    at scala.reflect.internal.Trees$$anonfun$itransform$2.apply(Trees.scala:1361)
    at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2600)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1360)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1258)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2589)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2587)
    at scala.collection.immutable.List.loop$1(List.scala:176)
    at scala.collection.immutable.List.mapConserve(List.scala:200)
    at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2587)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1404)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1258)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.reflect.api.Trees$Transformer.transformTemplate(Trees.scala:2563)
    at scala.reflect.internal.Trees$$anonfun$itransform$4.apply(Trees.scala:1408)
    at scala.reflect.internal.Trees$$anonfun$itransform$4.apply(Trees.scala:1407)
    at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2600)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1406)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1258)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2589)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:2587)
    at scala.collection.immutable.List.loop$1(List.scala:176)
    at scala.collection.immutable.List.mapConserve(List.scala:200)
    at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2587)
    at scala.reflect.internal.Trees$$anonfun$itransform$7.apply(Trees.scala:1426)
    at scala.reflect.internal.Trees$$anonfun$itransform$7.apply(Trees.scala:1426)
    at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2600)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1425)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
    at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:1258)
    at scala.tools.nsc.transform.Mixin$MixinTransformer.transform(Mixin.scala:468)
    at scala.tools.nsc.ast.Trees$Transformer.transformUnit(Trees.scala:147)
    at scala.tools.nsc.transform.Transform$Phase.apply(Transform.scala:30)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:467)
    at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:458)
    at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:467)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:425)
    at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1528)
    at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1513)
    at scala.tools.nsc.Global$Run.compileSources(Global.scala:1508)
    at scala.tools.nsc.Global$Run.compile(Global.scala:1609)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:130)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:105)
    at xsbt.CompilerInterface.run(CompilerInterface.scala:31)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sbt.internal.inc.AnalyzingCompiler.call(AnalyzingCompiler.scala:237)
    at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:111)
    at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:90)
    at org.jetbrains.jps.incremental.scala.local.IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:42)
    at org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:43)
    at org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:145)
    at org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:131)
    at org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:121)
    at org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:128)
    at org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:105)
    at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:63)
    at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319)

As I had no hint at all what caused the NPE it required some time to narrow it down. The NPE seems to be a bug in the scala compiler that was fixed in 2.13, see e.g. https://github.com/scala/bug/issues/8583 (please correct me if I am wrong in assuming a connection there?).

There still is some weirdness left in what actually caused this whole stuff to emerge (no, I did not expect to be able to substract DenseVectors from a Double xD). The operations in the initial code were a bit more complex, but essentially boil down to this line in the MWE:

value -= 0.5 * v.t * m * v

This should actually NOT produce a DenseVector at all imho!

Leaving the 0.5 aside, we multiply a transposed vector, a square matrix and a vector, all having the same dimension(s). This operation results in a scalar, not a vector!

However the 0.5 messes this up in the code (at least regarding the types) - multiplying a scalar on a transposed DenseVector (0.5 * v.t) doesn't result in a transposed DenseVector as expected, the result is of type DenseMatrix. This then causes the whole line to result in the incorrect type of DenseVector instead of Double.

Side Question: Why does the code show this behavior regarding the types? Anyone has an idea if this was a justified choice, or might this even be a bug in Breeze?

So it turns out our original code can easily be fixed by adding parantheses:

value -= 0.5 * (v.t * m * v)

correctly results in the addition of two Doubles, everything fine.

But the cause of the problem still puzzles me, I'm looking forward to receiving some helpful info :)

I just verified all four cases using `sbt compile` and the compiler behavior is completely identical. — Beztix, Nov 10 '20 at 08:59
I can replicate the errors, those are extremely weird. My humble hypothesis would be that **breeze** uses some fancy `macros` in order to optimize their operations which has the consequence of producing those weird errors. Sadly I do not know what else to say, maybe it would be worth to open an issue in **breeze**? — Luis Miguel Mejía Suárez, Nov 10 '20 at 14:41
Thanks for your input, I will do so as soon as I got the time for it (most likely tomorrow) and then keep this question updated if relevant info comes in — Beztix, Nov 10 '20 at 18:17
Breeze uses a lot of macros, but I don't think this is that. In particular, all Breeze macros related to operators like this are processed during the compilation of Breeze itself, basically for doing code generation. My guess is this is specialization related. — dlwh, Nov 10 '20 at 23:13

score 1 · Answer 1 · answered Nov 10 '20 at 23:12

First Question: This is an educated guess. Scala has somewhat weird handling of + and += because Scala lets you do (x: Any) + (y: String) and I've seen weird discrepancies in errors for different operators arising from that.

Second Question: It's a compiler bug and should probably be supported. You might try running with the -no-specialization flag in scalac and seeing if you get the same error. Specialization is used a lot in Breeze and I've stumbled on a lot of compiler bugs related to it over the years.

Side Question: It's a limitation/bug in Breeze. Scala parses .5 * v.t * m * v as 0.5.*(v.t).*(m).*(v) (i.e. left to right). v.t is a Transpose[DenseVector[Double]] (i.e. a row vector), and honestly the Transpose stuff isn't as fully baked as I would like. Historically we represented a row vector as a matrix and I suspect it's falling back to that, and so it sees matrix * matrix * vector which of course is a vector.

Using -= with incompatible types (Double and DenseVector) causes strange scala compiler error

1 Answers1