1

I have a spark application that contains multiple spark jobs to be run on Azure data bricks. I want to build and package the application into a fat jar. The application is able to compile successfully. While I am trying to package (command: sbt package) the application, it gives an error "[warn] multiple main classes detected: run 'show discoveredMainClasses' to see the list".

How to build the application jar (Without specifying any main class) so that I can upload it to Databricks job and specify the main classpath over there?

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Dharita Chokshi
  • 1,133
  • 3
  • 16
  • 39

1 Answers1

1

This message is just a warning (see [warn] in it), it doesn't prevent generation of the jar files (normal or fat). Then you can upload resulting jar to DBFS (or ADLS for newer Databricks Runtime versions), and create Databricks job either as Jar task, or Spark Submit task.

If sbt fails, and doesn't produce jars, then you have some plugin that forces error on the warnings.

Also notice that sbt package doesn't produce fat jar - it produce jar only for classes in your project. You will need to use sbt assembly (install sbt-assembly plugin for that) to generate fat jar, but make sure that you marked Spark & Delta dependencies as provided.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132