3

I have a Spark enviroment running on Ubuntu 16.2 over VirtualBox. Its configured to run locally and when I start Spark with

./start-all

I can access to it on VM via web-ui using the URL: http://localhost:8080

From the host machine (windows), I can access it too using the VM IP: http://192.168.x.x:8080.

enter image description here

The problem appears when I try to create a context from my host machine. I have a project in eclipse that uses maven, and I try to run the following code:

ConfigLoader.masterEndpoint = "spark://192.168.1.132:7077"

val conf = new SparkConf().setMaster(ConfigLoader.masterEndpoint).setAppName("SimpleApp")
val sc = new SparkContext(conf)

I got this error:

16/12/21 00:52:05 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.1.132:8080...
16/12/21 00:52:06 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.1.132:8080
org.apache.spark.SparkException: Exception thrown in awaitResult
    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)

I've tried changing the URL for:

ConfigLoader.masterEndpoint = "spark://192.168.1.132:7077"

Unsuccessfully.

Also, if I try to access directly to the master URL via web (http://localhost:7077 in VM), I don't get anything. I don't know if its normal.

What am I missing?

Community
  • 1
  • 1
dgcipp
  • 339
  • 3
  • 17

1 Answers1

4

In your VM go to spark-2.0.2-bin-hadoop2.7/conf directory and create spark-env.sh file using below command.

cp spark-env.sh.template spark-env.sh

Open spark-env.sh file in vi editor and add below line. 

SPARK_MASTER_HOST=192.168.1.132

Stop and start Spark using stop-all.sh and start-all.sh. Now in your program you can set the master like below.

val spark = SparkSession.builder()
  .appName("SparkSample")
  .master("spark://192.168.1.132:7077")
  .getOrCreate()
abaghel
  • 14,783
  • 2
  • 50
  • 66
  • Thanks!, this solves part of my problem. The other part was that I was not using the same versions of spark/scala betwen my App and the spark machine. This post helped me too:http://stackoverflow.com/questions/38559597/failed-to-connect-to-spark-masterinvalidclassexception-org-apache-spark-rpc-rp – dgcipp Dec 21 '16 at 23:44