Debugging SCollection contents when running tests

Question

Is there any way to view the contents of an SCollection when running a unit test (PipelineSpec)?

When running something in production on many machines there would be no way to see the entire collection in one machine, but I wonder is there a way to view the contents of an SCollection (for example when running a unit test in debug mode in intellij).

score 1 · Accepted Answer · answered Jan 08 '20 at 11:45

If you want to print debug statements to the console then you can use the debug method which is part of SCollections. A sample code shown below

    val stdOutMock = new MockedPrintStream
    Console.withOut(stdOutMock) {
      runWithContext { sc =>
        val r = sc.parallelize(1 to 3).debug(prefix = "===")
        r should containInAnyOrder(Seq(1, 2, 3))
      }
    }
    stdOutMock.message.filterNot(_ == "\n") should contain theSameElementsAs
      Seq("===1", "===2", "===3")

Debugging SCollection contents when running tests

1 Answers1