0

I have a problem with job-server-0.5.0 after upgraded DSE 4.6 to 4.7. If I run server_start.sh I'll get error "Failed to find Spark assembly in /usr/share/dse/spark/assembly/target/scala-2.10 You need to build Spark before running this program."

I found in /usr/share/dse/spark/bin/compute-classpath.sh

this code raises error

for f in ${assembly_folder}/spark-assembly*hadoop*.jar; do
  if [[ ! -e "$f" ]]; then
    echo "Failed to find Spark assembly in $assembly_folder" 1>&2
    echo "You need to build Spark before running this program." 1>&2
    exit 1
  fi
  ASSEMBLY_JAR="$f"
  num_jars=$((num_jars+1))
done

If I run /usr/share/dse/spark/bin/spark-submit I'll get this same error.

Filip Forszpaniak
  • 15
  • 6

1 Answers1

0

If you are using DSE you should most likely be launching the spark-jobserver without hitting compute-classpath. You can try modifying the launch script to use dse spark-submit like in the following example. 

# job server jar needs to appear first so its deps take higher priority
# need to explicitly include app dir in classpath so logging configs can be found
#CLASSPATH="$appdir:$appdir/spark-job-server.jar:$($SPARK_HOME/bin/compute-classpath.sh)"

#exec java -cp $CLASSPATH $GC_OPTS $JAVA_OPTS $LOGGING_OPTS $CONFIG_OVERRIDES $MAIN $conffile 2>&1 &
dse spark-submit --class $MAIN $appdir/spark-job-server.jar --driver-java-options "$GC_OPTS $JAVA_OPTS $LOGGING_OPTS" $conffile 2>&1 &

https://github.com/spark-jobserver/spark-jobserver/blob/f5406a50406c59f26c878d7cee7334d6b9203312/bin/server_start.sh

RussS
  • 16,476
  • 1
  • 34
  • 62
  • Thx, Now it's working in DSE 4.7 but in DSE 4.6 I had problem with it ;) Could you tell me, how I can stop job-server now? server_stop.sh doesn't stop the server. Should I kill it by e.g. ./bin/spark-class org.apache.spark.deploy.Client kill ? – Filip Forszpaniak May 28 '15 at 08:45
  • Killing the client sounds fine to me. The real solution is going to be to modify the script some more to fit whatever version you are using. I'm considering doing a better modification later when I have time. Dse 4.6 uses Spark 1.1.0 and the submit tool works differently so I would expect it to require a different command order to start. – RussS May 28 '15 at 15:22