It is possible to start an embedded instance of apache Spark node?

Question

I want to start an instance of a standalone Apache Spark cluster embedded into my java app. I tried to find some documentation at their website but not look yet.

Is this possible?

Stark industries FTW! Using Ironman clustering? – maasg Jun 25 '14 at 21:41 — maasg, Jun 25 '14 at 21:41
xD sorry mispell. Apache Spark – Rodrigo Jun 30 '14 at 14:18 — Rodrigo, Jun 30 '14 at 14:18

score 19 · Answer 1 · answered Oct 20 '14 at 01:58

19

You can create SparkContext in local mode, you just need to provide "local" as a spark master url to SparkConf

val sparkConf = new SparkConf().
  setMaster("local[2]").
  setAppName("MySparkApp")

val sc = new SparkContext(sparkConf)

answered Oct 20 '14 at 01:58

Eugene Zhulenev

9,714
2
30
40

score 9 · Answer 2 · edited Nov 20 '18 at 04:35

9

Yes -- you can use Spark in an embedded way with a "local" master.

SparkConf sparkConf = new SparkConf();//Create new spark config
sparkConf.setMaster("local[8]"); // local, using 8 cores (you can vary the number)
sparkConf.setAppName("MyApp");
SparkContext sc = new SparkContext(sparkConf);

This will run Spark within your JVM.

edited Nov 20 '18 at 04:35

Hasson

1,894
1
21
25

answered May 30 '16 at 18:31

Daniel Winterstein

2,418
1
29
41

Can Spark run in a multithreaded setting? I want to embed Spark in Jboss, but it seems that Spark can run only once in a JVM. – ps0604 May 18 '19 at 16:14

score 4 · Answer 3 · answered Oct 02 '20 at 18:28

Others answered this question but as for 2020 with Apache Spark version 3.0:

Java example:

SparkSession spark = SparkSession.builder().appName("Your app name").master("local[*]").getOrCreate();

master("local[*]") means run in a standalone mode with all available CPU cores.

Maven dependencies:

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.12</artifactId>
      <version>3.0.1</version>
      <scope>provided</scope>
    </dependency>
    
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.12</artifactId>
      <version>3.0.1</version>
    </dependency>

It is possible to start an embedded instance of apache Spark node?

3 Answers3