1

Anyone can tell me how to use jars and packages .

  1. I'm working on web aplication.
  2. For Engine side spark-mongo

bin/spark-submit --properties-file config.properties --packages org.mongodb.spark:mongo-spark-connector_2.11:2.4.1,com.crealytics:spark-excel_2.11:0.13.1 /home/PycharmProjects/EngineSpark.py 8dh1243sg2636hlf38m

  • I'm using above command but it's downloading each time from maven repository(jar & packages).
  • So now my concern is if i'm offline it gives me error
  • its good if their any way to download it only once so no need to download each time
  • any suggestion how to deal with it.
vishal
  • 25
  • 8

2 Answers2

1

Get all the jar files required then pass them as a parameter to the spark-submit.

This way you need not to download files everytime you submit the spark job.

You have to use --jars instead of --packages

bin/spark-submit --properties-file config.properties --jars /home/PycharmProjects/spark-excel_2.11-0.11.1.jar,/home/PycharmProjects/mongo-spark-connector_2.11-2.4.1.jar /home/PycharmProjects/EngineSpark.py 8dh1243sg2636hlf38m
Shubham Jain
  • 5,327
  • 2
  • 15
  • 38
  • from where i get jar files? – vishal Jun 12 '20 at 07:27
  • Download from maven repository directly from google – Shubham Jain Jun 12 '20 at 08:27
  • Exception in thread "main" org.apache.spark.SparkException: No main class set in JAR; please specify one with --class. – vishal Jun 12 '20 at 10:34
  • Can you post your exact command, there should be no space between jar files I updated my answer – Shubham Jain Jun 12 '20 at 10:42
  • bin/spark-submit --properties-file config.properties --jars /home/PycharmProjects/spark-excel_2.11-0.11.1.jar,/home/PycharmProjects/mongo-spark-connector_2.11-2.4.1.jar /home/PycharmProjects/EngineSpark.py 5ee357e38910242a7b9797c8 – vishal Jun 12 '20 at 11:25
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/215811/discussion-between-vishal-and-shubham-jain). – vishal Jun 12 '20 at 11:28
0

Exception in thread "main" org.apache.spark.SparkException: No main class set in JAR; please specify one with --class

It can be because it cannot find the jar file, maybe the path is incorrect or try to give --jars configuration at the end and --class first.

sudomudo
  • 74
  • 2
  • 13
  • bin/spark-submit --properties-file config.properties --jars /home/PycharmProjects/spark-excel_2.11-0.11.1.jar,/home/PycharmProjects/mongo-spark-connector_2.11-2.4.1.jar /home/PycharmProjects/EngineSpark.py 5ee357e38910242a7b9797c8 – vishal Jun 12 '20 at 11:25