2

I am working with Spark-shell for Scala and found a strange behaviour in Spark-shell REPL which is not there if i use any IDE.

I can declare the same immutable variable again and again in REPL, but the same is not allowed in IDE.

Here is the code in REPL:

scala> val rdd = sc.textFile("README.md")
rdd: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[5] at textFile at <console>:24

scala> val rdd = sc.textFile("README.md")
rdd: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[7] at textFile at <console>:24

scala> val rdd = sc.textFile("README.md")
rdd: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[9] at textFile at <console>:24

scala>     val rdd = sc.textFile("README.md")
rdd: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[11] at textFile at <console>:24

And here is the same thing I am trying in Eclipse IDE, and it shows compile time error:

enter image description here

Is there anything i am missing any configuration for Spark-shell REPL?

Or is it the expected behaviour?

Community
  • 1
  • 1
KayV
  • 12,987
  • 11
  • 98
  • 148
  • 3
    Possible duplicate of [Why is it possible to declare variable with same name in the REPL?](https://stackoverflow.com/questions/22772320/why-is-it-possible-to-declare-variable-with-same-name-in-the-repl) - this is actually a feature of the scala REPL. – Oli Mar 28 '19 at 09:45

1 Answers1

1

In your REPL, your code is actually translated as follows:

object SomeName {
  val rdd = sc.textFile("README.md")
}
object Some_Other_Name {
  val rdd = sc.textFile("README.md")
}

Since both your rdd vals are defined in separate Singletons, there is no name collision between them. and since this happens behind the scenes in the REPL, you feel as if you are making reassignments to the same val.

In an IDE, all our code is written inside Classes or Scala singletons(Objects), so in the same scope (inside the resp. Class/Object) you are returned an error.

Yayati Sule
  • 1,601
  • 13
  • 25