How to add Spark to Maven project in Eclipse?

Question

I would like to start Spark project in Eclipse using Maven. I've installed m2eclipse and I have a working HelloWorld Java application in my Maven project.

I would like to use Spark framework and I'm following directions from the official site. I've added Spark repository to my pom.xml:

<repository>
      <id>Spark repository</id>
      <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
</repository>

And then the dependency:

<dependency>
      <groupId>spark</groupId>
      <artifactId>spark</artifactId>
      <version>0.9.9.4-SNAPSHOT</version>
</dependency>

But I'm getting an error in Eclipse:

Missing artifact spark:spark:jar:0.9.9.4-SNAPSHOT

How can I resolve this issue? I don't want to download Spark's jar file and place in the local repository.

This is my pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.myproject</groupId>
  <artifactId>Spark1</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>Spark1</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <repository>
      <id>Spark repository</id>
      <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
  </repository>

  <dependencies>
<!--     (...) -->

    <dependency>
      <groupId>spark</groupId>
      <artifactId>spark</artifactId>
      <version>0.9.9.4-SNAPSHOT</version>
    </dependency>

  </dependencies>

</project>

Could you show us your whole `pom.xml`? You can omitt other dependencies but I'd like to see the structure. Also make sure your Eclipse project really has Maven enabled (rigt click on project=>`Configure`=>`Convert to Maven Project`). — André Stannek, Mar 04 '13 at 21:16

Sumit Ramteke · Answer 1 · 2017-07-03T11:34:47.223

12

Currently no repository is required to add for Spark library loading

You just need to add

<dependency>
    <groupId>com.sparkjava</groupId>
    <artifactId>spark-core</artifactId>
    <version>2.6.0</version>
</dependency>

And that's it.

Useful tutorials to play with is here

edited Jul 03 '17 at 11:34

answered Aug 04 '13 at 12:07

Sumit Ramteke

1,487
1
16
39

score 7 · Accepted Answer · answered Mar 04 '13 at 22:02

7

The repository block needs to be wrapped in a repositories block:

<repositories>
    <repository>
        <id>Spark repository</id>
        <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
    </repository>
</repositories>

answered Mar 04 '13 at 22:02

André Stannek

7,773
31
52

score 5 · Answer 3 · answered Aug 31 '14 at 08:32

Reason for failure is 0.9.9.4-SNAPSHOT is not available.Below is the list of the snapshots available. Use one among them based on your requirement.

0.9.8-SNAPSHOT/ Sat May 21 21:54:23 UTC 2011
0.9.9-SNAPSHOT/ Mon May 23 10:57:38 UTC 2011
0.9.9.1-SNAPSHOT/ Thu May 26 09:47:03 UTC 2011
0.9.9.3-SNAPSHOT/ Thu Sep 01 07:53:59 UTC 2011

Thanks, Sankara Reddy

score 0 · Answer 4 · answered May 24 '13 at 12:37

0

I had run into same issue because initially I started with different repository url for spark and then to use earlier version I changed the repository url. Some how it didn't seem to come into effect until I changed the repository id. Try changing the repository id.
Could be bug in maven because running maven from console also couldn't resolve the dependency without updating the id.

answered May 24 '13 at 12:37

ptsering

23
4

This actually worked for me.. all I did was I changed id from "Spark repository" to "Spark Repository". – FSP May 27 '13 at 02:47

score 0 · Answer 5 · answered May 27 '14 at 20:27

please add the repository , tag inside the repositories tag like below

<repositories>
        <repository>
            <id>Spark repository</id>
            <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
        </repository>
    </repositories>

score 0 · Answer 6 · answered Jan 18 '15 at 23:38

The last versions (2.1 and later) of Spark only need the dependency defined inside the pom.xml file

<dependency>
    <groupId>com.sparkjava</groupId>
    <artifactId>spark-core</artifactId>
    <version>2.1</version>
</dependency>

the repository definition is not longer required

score 0 · Answer 7 · answered Feb 15 '16 at 08:00

0

use this latest repository. http://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10/1.6.0

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.6.0</version>
</dependency>

answered Feb 15 '16 at 08:00

spartan

88
1
7

score 0 · Answer 8 · answered Oct 13 '16 at 16:00

use this, and also make sure you change spark library to version 2.11.x in eclipse project build path

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.spark-scala</groupId>
    <artifactId>spark-scala</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>${project.artifactId}</name>
    <description>Spark in Scala</description>
    <inceptionYear>2010</inceptionYear>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.tools.version>2.10</scala.tools.version>
        <!-- Put the Scala version of the cluster -->
        <scala.version>2.10.4</scala.version>
    </properties>

    <!-- repository to add org.apache.spark -->
    <repositories>
        <repository>
            <id>cloudera-repo-releases</id>
            <url>https://repository.cloudera.com/artifactory/repo/</url>
        </repository>
    </repositories>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <!-- see http://davidb.github.com/scala-maven-plugin -->
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.1</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.13</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <includes>
                        <include>**/*Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>

            <!-- "package" command plugin -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>1.2.1</version>
        </dependency>
    </dependencies>
</project>

How to add Spark to Maven project in Eclipse?

8 Answers8