Any examples of code integrating Scalatra with Spark

Question

I'm new to Scalatra. I've been experimenting with Spark. I want to integrate Spark with a Scalatra web interface. I have found two examples so far of code integrating Scalatra with Spark. One is from Github (link below) and does not seem to follow the normal build configuration per Scalatra's documentation, and so I'm not able to create a new Scalatra-Spark project using that model.

https://github.com/ytaras/scalate_spark

My own attempt to integrate a Simple Spark operation is getting the following error:

org.eclipse.jetty.server.AbstractConnector: method <init>()V not found
java.lang.NoSuchMethodError: org.eclipse.jetty.server.AbstractConnector: method <init>()V not found

My code that is throwing the error:

import org.apache.spark._
import org.apache.spark.SparkContext._

import scala.math.random

/** Computes an approximation to pi */
object SparkPi {
    lazy val sc = new SparkContext("local", "Spark Pi")

    def calcPi(slices: Int = 2) {

        println("SparkPi.calcPi - sc: " + sc)
        val n = 100000 * slices

        val count = sc.parallelize(1 to n, slices).map { i =>
          val x = random * 2 - 1
          val y = random * 2 - 1
          if (x * x + y * y < 1) 1 else 0
        }.reduce(_ + _)

        println("Pi is roughly " + 4.0 * count / n)    
    }
}

I'm not sure how to best indicate my Scalatra project structure, which is probably very relevant here for anyone trying to help me with this problem. But I'm using the normal Scalatra project creation per their documentation. I'm calling this SparkPi class through a normal Scalatra Servlet route:

get("/pi") {
    calcPi()
}

score 1 · Answer 1 · answered Oct 29 '14 at 20:16

This happens when the wrong jetty library is referenced as a dependency. Spark 1.1.0 supports Jetty version 7.

In your build.scala file your dependency tree for jetty-webapp may include the org.eclipse.jetty:jetty-servlet assembly as a version other than 7.6.9.v20130131. Spark 1.1.0 relies on this version of the jetty-servlet.

The solution here is to create an exclusion for jetty-servlet for jetty-webapp, although that may cause issues elsewhere in your application.

score 0 · Answer 2 · answered Feb 24 '15 at 20:11

I faced the same error with Spark 1.2.1 and Scalatra 2.3.0. I did end up getting Scalatra and Spark working together, I had to do the following:

Change the Scala version to 2.10.4 (line 13 in project/build.scala). The Spark docs say "Spark 1.2.0 uses Scala 2.10. To write applications in Scala, you will need to use a compatible Scala version (e.g. 2.10.X)." 2.10.4 is the latest 2.10.X version available.
Change the Jetty version to 8.1.14.v20131031 (in the libraryDependencies section of build.scala). This is to match the Jetty dependency listed in the Spark 1.2.1 Maven Package

... and it worked! Woohoo!

Here's the complete working project on GitHub, which defines a JSON REST API with Scalatra that invokes Spark code:

https://github.com/curran/scalatra-spark

Any examples of code integrating Scalatra with Spark

2 Answers2