scala code throw exception in spark

Question

I am new to scala and spark. Today I tried to write some code, and let it run on spark, but got an exception.

this code work in local scala

import org.apache.commons.lang.time.StopWatch
import org.apache.spark.{SparkConf, SparkContext}

import scala.collection.mutable.ListBuffer
import scala.util.Random

  def test(): List[Int] = {
    val size = 100
    val range = 100
    var listBuffer = new ListBuffer[Int] // here throw an exception
    val random = new Random()
    for (i <- 1 to size)
      listBuffer += random.nextInt(range)
    listBuffer.foreach(x => println(x))
    listBuffer.toList
  }

but when I put this code into spark, it throw an exception says:

15/01/01 14:06:17 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ObjectRef.create(Ljava/lang/Object;)Lscala/runtime/ObjectRef;
    at com.tudou.sortedspark.Sort$.test(Sort.scala:35)
    at com.tudou.sortedspark.Sort$.sort(Sort.scala:23)
    at com.tudou.sortedspark.Sort$.main(Sort.scala:14)
    at com.tudou.sortedspark.Sort.main(Sort.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

if I comment out the below code, the code work in spark

for (i <- 1 to size)

can someone explain why, please.

What is line 35 of Sort.scala? The `new ListBuffer` line? It seems odd that that throws an exception which goes away when you remove the for later. Please post a complete example that fails with spark — The Archetypal Paul, Jan 01 '15 at 08:42
This looks like a scala version mismatch. Check that the version of scala you're using matches that that spark was built against, and the one on the cluster where you're running. — lmm, Jan 01 '15 at 09:45

score 13 · Accepted Answer · edited Feb 10 '16 at 19:32

13

Thanks @Imm, I have solved this issue. The root cause is that my local scala is 2.11.4, but my spark cluster is running at 1.2.0 version. The 1.2 version of spark was compiled by 2.10 scala.

So the solution is compile local code by 2.10 scala, and upload the compiled jar into spark. Everything works fine.

edited Feb 10 '16 at 19:32

gsamaras

71,951
46
188
305

answered Jan 04 '15 at 06:15

Chan

914
9
25

Can you explain what exactly did you do? Just setting `scalaVersion := "2.10.0"` from `2.11.4` in the .sbt file? – gsamaras Feb 10 '16 at 20:02
I have the same question as @gsamaras did – HappyCoding Feb 15 '16 at 09:15
@HappyCoding I do not remember how I solved it, but here is my experience: https://gsamaras.wordpress.com/code/spark-cluster/ – gsamaras Feb 15 '16 at 09:37
thanks! I solved the problem by following your suggestion, downgrade scala compiler version. – HappyCoding Feb 16 '16 at 02:12

scala code throw exception in spark

1 Answers1

Linked