Spark Job-Server configuaration in StandAlone cluster

Question

I am trying to set up a Spark JobServer (SJS) to execute jobs on a Standalone Spark cluster. I am trying to deploy SJS on one of the non-master nodes of SPARK cluster. I am not using the docker, but trying to do manually.

I am confused with the help documents in SJS github particulary the deployment section. Do I need to edit both local.conf and local.sh to run this?

Can someone point out the steps to set up the SJS in the spark cluster?

Thanks! Kiran

Update: I created a new environment to deploy jobserver in one of the nodes of the cluster: Here are the details of it:

env1.sh:

DEPLOY_HOSTS="masked.mo.cpy.corp"
APP_USER=kiran
APP_GROUP=spark
INSTALL_DIR=/home/kiran/job-server
LOG_DIR=/var/log/job-server
PIDFILE=spark-jobserver.pid
JOBSERVER_MEMORY=1G
SPARK_VERSION=1.6.1
MAX_DIRECT_MEMORY=512M
SPARK_HOME=/home/spark/spark-1.6.1-bin-hadoop2.6
SPARK_CONF_DIR=$SPARK_HOME/conf
SCALA_VERSION=2.11.6

env1.conf

spark {

  master = "local[1]"
  webUrlPort = 8080
  job-number-cpus = 2

  jobserver {
    port = 8090
    bind-address = "0.0.0.0"
    jar-store-rootdir = /tmp/jobserver/jars
    context-per-jvm = false
    jobdao = spark.jobserver.io.JobFileDAO
    filedao {
      rootdir = /tmp/spark-job-server/filedao/data
    }
    datadao {
      rootdir = /tmp/spark-jobserver/upload
    }

    result-chunk-size = 1m
  }

  context-settings {
    num-cpu-cores = 1
    memory-per-node = 1G
  }
  home = "/home/spark/spark-1.6.1-bin-hadoop2.6"
}

score 1 · Answer 1 · answered Jun 09 '16 at 03:25

1

Why don't you set JOBSERVER_FG=1 and try running server_start.sh, this would run the process in foreground and should display the error to stderr.

answered Jun 09 '16 at 03:25

noorul

1,283
1
8
18

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

Yes, you have edit both files adapting them for your cluster.

The deploy steps are explained below:

Copy config/local.sh.template to <environment>.sh and edit as appropriate.

This file is mostly for environment variables that are used by the deployment script and by the server_start.sh script. The most important ones are: deploy host (it's the ip or hostname where the jobserver will be run), user and group of execution, JobServer memory (it will be the driver memory), spark version and spark home.

Copy config/shiro.ini.template to shiro.ini and edit as appropriate. NOTE: only required when authentication = on

If you are going to use shiro authentication, then you need this step.

Copy config/local.conf.template to <environment>.conf and edit as appropriate.

This is the main configuration file for JobServer and for the contexts that JobServer will create. The full list of the properties you can set in this file can be seen on this link.

bin/server_deploy.sh <environment>

After editing the configuration files, you can deploy using this script. The parameter must be the name that you chose for your .conf and .sh files.

Once you run the script, JobServer will connect to the host entered in the .sh file and will create a new directory with some control files. Then, every time you need to change a configuration entry, you can do it directly on the remote machine: the .conf file will be there with the name you chose and the .sh file will be renamed to settings.sh.

Please note that, if you haven't configured an SSH key based connection between the machine where you run this script and the remote machine, you will be prompted for password during its execution.

If you have problems with the creation of directories on the remote machine, you can try and create them yourself with mkdir (they must match the INSTALL_DIR configuration entry of the .sh file) and change their owner user and group to match the ones entered in the .sh configuration file.

On the remote server, start it in the deployed directory with server_start.sh and stop it with server_stop.sh

This is very informative. Once you have done all other steps, you can start JobServer service on the remote machine by running the script server_start.sh and you can stop it with server_stop.sh

Thanks @Daniel-de-Paula for the detailed steps. But I see the job server is not starting inspite of this. I have included the env1.sh and env1.conf in the issue description above. On starting the server_start.sh, I see nothing is happening. I can see a new PID is started for job server. But the port 8090 is not opened up. Hence all curl request fails to the server. Any idea what is missing? — Kiran, Jun 01 '16 at 18:11
@Kiran are you starting the server_start.sh in the target machine (i.e. the one you configured in the first lines of the `.sh` file)? Maybe you can debug by looking at `$LOG_DIR/jobserver` (defaults to `/var/log/jobserver`) — Daniel de Paula, Jun 01 '16 at 18:21
@Daneil-de-Paula: Yes, I am starting it in the target machine. This is what I did from beginning: In my local system sbt built the job server with env1.conf and env1.sh. Deployed from local system to the server using server_deploy.sh. In target system, started the spark master and a slave instance. can see the cluster running in 8080 port. Now tried to start the job server expecting port 8090 will be open. But netstat -l -n does not show 8090 port. But I see a new process running with a new PID for the jobserver. The jobserver debug file does not specify if something has gone wrong. — Kiran, Jun 01 '16 at 18:43
@Kiran not sure if it's related, but you should change the `master` configuration item in the `.conf` file to the spark's master url, e.g. `master = "spark://hostname:7077"` — Daniel de Paula, Jun 01 '16 at 18:45
@Daneil: I tried it.. But no luck.. there is still no port for 8090.. On `server_start.sh` , I can see a new process for spark created and a listening port 9999 opens up, but nothing for 8090... I see this method somewhat broken.. Do you have any other suggestion on how to setup the job server without the docker? — Kiran, Jun 01 '16 at 19:23
@Kiran I see. I believe the deploy script is the recommended procedure if you don't want to use docker. I'm afraid I won't be able to help you further, but you can talk to the developers in [JobServer's Gitter](https://gitter.im/spark-jobserver/spark-jobserver), they are always willing to help and they know better than me. You can even post a link to this question there, so you will have more visibility for your problem and more people will be able to help. — Daniel de Paula, Jun 01 '16 at 20:20

Spark Job-Server configuaration in StandAlone cluster

2 Answers2