0

I have an existing spark-job, the functionality of this spark-job is to connect kafka-server get the data and then storing the data into cassandra tables, now this spark-job is running on server inside spark-2.1.1-bin-hadoop2.7/bin but whenever I am trying to run this spark-job from other location, Its not running, this spark-job contains some JavaRDD related code. Is there any chance, I can run this spark-job from outside also by adding any dependency in pom or something else?

Sat
  • 3,520
  • 9
  • 39
  • 66
  • 2
    How are you running your job? you can run spark-submit from anywhere. – banjara May 22 '17 at 11:40
  • Yes I am using spark-submit... I am not aware of this that much.. can you pls tell me little bit clearly, actually my seniors are running this, I don't have that much knowledge on this... Can i directly copy spar-submit & my jar into another folder(Ex:- test) and run there directly ? – Sat May 22 '17 at 11:43
  • Can you explain _"whenever I am trying to run this spark-job from other location, Its not running"_? What are the errors? – Jacek Laskowski May 22 '17 at 11:51
  • I think you are specifying relative paths, if you use absolute path it will work from other directories as well. – banjara May 22 '17 at 11:59
  • Where do you run your spark cluster standalone, yarn, mesos ? – FaigB May 22 '17 at 12:07
  • @JacekLaskowski errors :- `Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at debug.KafkaToCassandraJob.main(KafkaToCassandraJob.java:67) Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ' – Sat May 23 '17 at 11:49

1 Answers1

0

whenever I am trying to run this spark-job from other location, Its not running

spark-job is a custom launcher script for a Spark application, perhaps with some additional command-line options and packages. Open it, review the content and fix the issue.

If it's too hard to figure out what spark-job does and there's no one nearby to help you out, it's likely time to throw it away and replace with the good ol' spark-submit.

Why don't you use it in the first place?!

Read up on spark-submit in Submitting Applications.

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420