Connecting to remote master on standalone Spark

Question

I launch Spark in standalone mode on my remote server via following next steps:

cp spark-env.sh.template spark-env.sh
append to spark-env.sh SPARK_MASTER_HOST=IP_OF_MY_REMOTE_SERVER
and run next commands for standalone mode: sbin/start-master.sh sbin/start-slave.sh spark://IP_OF_MY_REMOTE_SERVER:7077

And I try to connect to remote master:

val spark = SparkSession.builder()
  .appName("SparkSample")
  .master("spark://IP_OF_MY_REMOTE_SERVER:7077")
  .getOrCreate()

And I receive the following errors:

ERROR SparkContext: Error initializing SparkContext.
java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries!

and warnings:

    WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078.
.....
    WARN Utils: Service 'sparkMaster' could not bind on port 7092. Attempting port 7092.

Are you tryng to run spark in LOCAL-Standalone mode? With all nodes on one machine? Do you have other Spark-Masters running already which use these ports? — Rick Moritz, Aug 23 '17 at 11:43
Can you put the `spark-submit`command you re using to submit you job? — user1314742, Aug 25 '17 at 10:26

score 6 · Accepted Answer · answered Aug 24 '17 at 04:47

I advise against submitting spark jobs remotely using the port opening strategy, because it can create security problems and is in my experience, more trouble than it's worth, especially due to having to troubleshoot the communication layer.

Alternatives:

1) Livy - now an Apache project! http://livy.io or http://livy.incubator.apache.org/

2) Spark Job server - https://github.com/spark-jobserver/spark-jobserver

Similar Q&A: Submitting jobs to Spark EC2 cluster remotely

If you insist on connecting without libraries like Livy, then opening the ports to ensure communication is required. The Spark network comm docs: http://spark.apache.org/docs/latest/security.html#configuring-ports-for-network-security

Since you're not using YARN (per your Standalone design), the prior link to YARN remote submission may not be relevant.

score 1 · Answer 2 · answered Aug 21 '17 at 13:03

The Spark Documentation says

spark.driver.port

(random)    Port for the driver to listen on. This is used for communicating with the executors and the standalone Master.

spark.port.maxRetries

16  Maximum number of retries when binding to a port before giving up. When a port is given a specific value (non 0), each subsequent retry will increment the port used in the previous attempt by 1 before retrying. This essentially allows it to try a range of ports from the start port specified to port + maxRetries.

You need to ensure that the Spark Master is running on remote host at port 7077. Also the firewall must allow connections to it.

AND

Also, you need to copy core-site.xml file from your cluster to HADOOP_CONF_DIR, so that Spark service can read hadoop settings, such as the IP address of your master. Read here for more..

Hope it helps!

I specified port as 7077 and opened firewall restrictions – pacman Aug 21 '17 at 13:41 — pacman, Aug 21 '17 at 13:41
Are you using an IaaS environment ? – nitinr708 Aug 21 '17 at 13:53 — nitinr708, Aug 21 '17 at 13:53

score 1 · Answer 3 · answered Aug 28 '17 at 09:12

The spark-jobserver seems very tempting but has some issues. I'd recommend the "hidden" spark REST api! It's not documented but it's super easy and much more comfortable, Unlike jobserver which requires maintenance (another thing you need to worry about and troubleshoot - and it has it's problems) Also you have great library for that - https://github.com/ywilkof/spark-jobs-rest-client

Connecting to remote master on standalone Spark

3 Answers3