Subtract an RDD from another RDD doesn't work correctly

Question

I want to subtract an RDD from another RDD. I looked into the documentation and I found that subtract can do that. Actually, when I tested subtract, the final RDD remains the same and the values are not removed!

Is there any other function to do that? Or am I using subtract incorrectly?

Here is the code that I used:

 val vertexRDD: org.apache.spark.rdd.RDD[(VertexId, Array[Int])]
 val clusters  = vertexRDD.takeSample(false, 3)
 val clustersRDD: RDD[(VertexId, Array[Int])] = sc.parallelize(clusters)
 val final = vertexRDD.subtract(clustersRDD)
 final.collect().foreach(println(_))

Ophir Yoktan · Accepted Answer · 2015-06-14T14:57:58.203

3

Performing set operations like subtract with mutable types (Array in this example) is usually unsupported, or at least not recommended.

Try using a immutable type instead.

I believe WrappedArray is the relevant container for storing arrays in sets, but i'm not sure.

edited Jun 14 '15 at 14:57

answered Jun 14 '15 at 14:51

Ophir Yoktan

8,149
7
58
106

I need to use an Array ! How can I fix that ? Or what is the other type that I can use ? Otherwise, I think that Array is an immutable type in Scala. – Ronald Segan Jun 14 '15 at 14:55
@OmarMasmoudi No, `Array` is just the same mutable JVM array as in other languages. The only difference is, it is handled not covariantly in Scala (it is invariant in Scala). – Gábor Bakos Jun 14 '15 at 15:08
@OphirYoktan Ah okay ! Thank you ! It works when I tested with Seq ! – Ronald Segan Jun 14 '15 at 15:26

score 2 · Answer 2 · edited May 18 '16 at 09:35

2

If your rdd is composed of mutables object it wont work... problem is it wont show an error either so this kind of problems are hard to identify, i had a similar one yesterday and i used a workaround.

rdd.keyBy( someImmutableValue ) -> do this using the same key value to
 both your rdds

val resultRDD = rdd.subtractByKey(otherRDD).values

edited May 18 '16 at 09:35

ρяσѕρєя K

132,198
53
198
213

answered May 18 '16 at 09:17

Felix

140
10

score 1 · Answer 3 · edited Jan 12 '18 at 21:51

Recently I tried the subtract operation of 2 RDDs (of array List) and it is working. The important note is - the RDD val after .subtract method should be the list from where you're subtracting, not the other way around.

Correct: val result = theElementYouWantToSubtract.subtract(fromList)

Incorrrect: val reuslt = fromList.subtract(theElementYouWantToSubtract) (will not give any compile/runtime error message)

Subtract an RDD from another RDD doesn't work correctly

3 Answers3

Linked