Mock a Spark RDD in the unit tests

Question

Is it possible to mock a RDD without using sparkContext?

I want to unit test the following utility function:

 def myUtilityFunction(data1: org.apache.spark.rdd.RDD[myClass1], data2: org.apache.spark.rdd.RDD[myClass2]): org.apache.spark.rdd.RDD[myClass1] = {...}

So I need to pass data1 and data2 to myUtilityFunction. How can I create a data1 from a mock org.apache.spark.rdd.RDD[myClass1], instead of create a real RDD from SparkContext? Thank you!

Why could you not? Those are just basic objects, you can mock them, though most probably it is usually easier to use `sc.parallelize` than mock the `map`/`flatMap`/... methods too. — Gábor Bakos, Jun 19 '15 at 20:23

score 21 · Answer 1 · answered Jun 19 '15 at 21:14

21

RDDs are pretty complex, mocking them is probably not the best way to go about creating test data. Instead I'd recommend using sc.parallelize with your data. I'm also (somewhat biased) think that https://github.com/holdenk/spark-testing-base can help by providing a trait to setup & teardown the Spark Context for your tests.

answered Jun 19 '15 at 21:14

Holden

7,392
1
27
33

it seems that spark-testing-base latest version is for Spark 1.5 - are there any options for testing with Spark 2+ ? – oDDsKooL Jan 17 '17 at 15:06
There are, the current release supports 2.1 – Holden Apr 03 '17 at 07:32

score 14 · Accepted Answer · edited Jun 20 '20 at 09:12

I totally agree with @Holden on that!

Mocking RDDS is difficult; executing your unit tests in a local Spark context is preferred, as recommended in the programming guide.

I know this may not technically be a unit test, but it is hopefully close enough.

Unit Testing

Spark is friendly to unit testing with any popular unit test framework. Simply create a SparkContext in your test with the master URL set to local, run your operations, and then call SparkContext.stop() to tear it down. Make sure you stop the context within a finally block or the test framework’s tearDown method, as Spark does not support two contexts running concurrently in the same program.

But if you are really interested and you still want to try mocking RDDs, I'll suggest that you read the ImplicitSuite test code.

The only reason they are pseudo-mocking the RDD is to test if implict works well with the compiler, but they don't actually need a real RDD.

def mockRDD[T]: org.apache.spark.rdd.RDD[T] = null

And it's not even a real mock. It just creates a null object of type RDD[T]

Mock a Spark RDD in the unit tests

2 Answers2

Linked