1

I have a Spark based Kubernetes cluster where I am using spark-submit to submit the jobs on cluster as needed. e.g.

spark-submit \
    --master spark://my-spark-master-svc:7077 \
    --class com.Main \
    examples/jars/my-spark-application.jar

Here I have uploaded file my-spark-application.jar using kubectl cp under the directory examples/jars on the master Pod/container before I run the spark-submit command.

Another option could be by mounting a Volume on the cluster and share the jar on the volume that way.

What is the typical way to share the application jar with the spark cluster while using spark-submit on Kubernetes?

adesai
  • 370
  • 3
  • 22
  • I'm not sure whether there is a typical way, but it might be interesting to you to hear how I did it on a previous project. We used artifactory, which is a repository on which you can store any kind of artifacts (files, binaries, ...) that was used for many parts of the projects. We had a CICD pipeline that automatically shipped our jars on there. In our `spark-submit` command, we then simply referred to the URL of our artifactory like so: `http://artifactory-url:service-port/path-to-jar/jar-name.jar`. – Koedlt Nov 25 '22 at 09:02

0 Answers0