3

I create sbt project with Intellij and build Artifacts to jar file. I put jar file to server and submit, but I got this error:

spark-submit --master spark://master:7077 --class streaming_process spark-jar/spark-streaming.jar

Error: Failed to load class streaming_process.
21/01/23 04:41:32 INFO ShutdownHookManager: Shutdown hook called
21/01/23 04:41:32 INFO ShutdownHookManager: Deleting directory /tmp/spark-982e8fe3-9421-45bd-aced-e46c4d756054

My code

// Code Block 2 Starts Here
    val spark = SparkSession.builder
      .master("spark://master:7077")
      .appName("Stream Processing Application")
      .getOrCreate()

    spark.sparkContext.setLogLevel("ERROR")
    // Code Block 2 Ends Here

    // Code Block 3 Starts Here
    // Stream meetup.com RSVP Message Data from Kafka
    val meetup_rsvp_df = spark.readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", kafka_bootstrap_servers)
      .option("subscribe", kafka_topic_name)
      .option("startingOffsets", "latest")
      .load()

You can see my project image:

enter image description here

blackbishop
  • 30,945
  • 11
  • 55
  • 76
Truong-Giang Nguyen
  • 81
  • 2
  • 8

1 Answers1

0

JVM can't find the jar which contains streaming_process class. Please use --jars spark-jar/spark-streaming.jar option.

Salim
  • 2,046
  • 12
  • 13
  • I tried but got the same issue `giangnt@node2:~/spark-jar$ spark-submit --master spark://master:7077 --class streaming_process --jars spark-streaming.jar, spark-sql-kafka-0-10_2.12-3.0.1.jar Error: Failed to load class streaming_process. 21/01/23 15:56:38 INFO ShutdownHookManager: Shutdown hook called 21/01/23 15:56:38 INFO ShutdownHookManager: Deleting directory /tmp/spark-587836d9-1709-4c70-b549-34081d400f21 giangnt@node2:~/spark-jar$` – Truong-Giang Nguyen Jan 23 '21 at 08:57
  • spark-submit --master spark://master:7077 --class streaming_process --jars spark-sql-kafka-0-10_2.12-3.0.1.jar spark-streaming.jar – Salim Jan 24 '21 at 20:40