0

I'm struggling to understand why implicit imports do not work as I expect them in scalatest. The simplified failing example (using spark, but I can make it fail with my custom class also) is as follows:

class FailingSpec extends FlatSpec with Matchers with MySparkContext {

    val testSqlctx = sqlctx
    import sqlctx.implicits._

    "sql context implicts" should "work" in {
        val failingDf = Seq(ID(1)).toDS.toDF
    }
}

The MySparkContext trait creates and destroys spark context in beforeAll and afterAll, and makes sqlctx available (already having to reassign it to a local variable in order to import implicits is a puzzle, but maybe for a different time). The .toDS and .toDF are then implicit methods imported from sqlctx.implicits. Running the test results in a java.lang.NullPointerException.

If I move import into test block things work:

class WorkingSpec extends FlatSpec with Matchers with MySparkContext {

    "sql context implicts" should "work" in {      
        val testSqlctx = sqlctx
        import sqlctx.implicits._

        val workingDf = Seq(ID(1)).toDS.toDF
    }
}

Any ideas why can't I import implicits at the top level of the test class?

psarka
  • 1,562
  • 1
  • 13
  • 25

1 Answers1

1

beforeAll runs before any of the tests, but does not run before the constructor for the class. The order of operations in the first snippet is:

  1. Constructor invoked, executing val testSqlctx = sqlctx and import sqlctx.implicits._

  2. beforeAll invoked

  3. Tests run

And the order of operations for the second snippet:

  1. beforeAll invoked

  2. Tests run, executing val testSqlctx = sqlctx and import sqlctx.implicits._

Assuming you give your SparkContext a default (null) value and initialize it in beforeAll, the first order of operations would try to use sqlctx when it is still null, thus causing the null pointer exception.

Alfredo Gimenez
  • 2,174
  • 1
  • 14
  • 19
  • Thanks, so obvious in hindsight! So I guess I should initialize things properly in the constructor of the trait, rather than in the `beforeAll` method. – psarka Aug 29 '16 at 18:52