1

By defining different ports I was hoping to launch 2 masters on the same machine, but it failed miserably:

$ $SPARK_HOME/sbin/start-master.sh --port 8001 --webui-port 8011
starting org.apache.spark.deploy.master.Master, logging to /Users/brandl/bin/spark-2.2.0-bin-hadoop2.7/logs/spark-brandl-org.apache.spark.deploy.master.Master-1-scicomp-mac-12.local.out
$ $SPARK_HOME/sbin/start-master.sh --port 8002 --webui-port 8012
org.apache.spark.deploy.master.Master running as process 29436.  Stop it first.

Why isn't that working? Do I need to adjust more settings to allow for multiple instances?

Following the same logic I may want to start two workers on the same machine. But even if they are supposed to be connected to different masters this fails as well with similar error:

$SPARK_HOME/sbin/start-slave.sh --webui-port 8050 spark://foo:7077
starting org.apache.spark.deploy.worker.Worker, logging to /Users/brandl/bin/spark-2.2.0-bin-hadoop2.7/logs/spark-brandl-org.apache.spark.deploy.worker.Worker-1-scicomp-mac-12.local.out
starting org.apache.spark.deploy.worker.Worker, logging to /Users/brandl/bin/spark-2.2.0-bin-hadoop2.7/logs/spark-brandlorg.apache.spark.deploy.worker.Worker-2-scicomp-mac-12.local.out
$SPARK_HOME/sbin/start-slave.sh --webui-port 8051 spark://bar:7077
org.apache.spark.deploy.worker.Worker running as process 29503.  Stop it first.
org.apache.spark.deploy.worker.Worker running as process 29526.  Stop it first.

If checked the spark standalone docs but could not find any guidance about what I'm doing wrong.

Holger Brandl
  • 10,634
  • 3
  • 64
  • 63

1 Answers1

7

Since there was no answer in here, I went back to the spark sources, and searched for the code bits where the error was thrown. This revelaled that by setting a different SPARK_IDENT_STRING multiple masters and workers can run on the same machine

$ $SPARK_HOME/sbin/start-master.sh --port 8001 --webui-port 8011
export SPARK_IDENT_STRING=foo
$ $SPARK_HOME/sbin/start-master.sh --port 8002 --webui-port 8012

The default value is the username ($USER)

Holger Brandl
  • 10,634
  • 3
  • 64
  • 63