How to add library to spark shell

Question

I have a library that I want to use in spark shell, how can I add this library to be accessible from the spark shell? sbt :

resolvers += Resolver.bintrayRepo("unsupervise", "maven")
libraryDependencies += "com.github.unsupervise" %% "spark-tss" % "0.1.1"

maven :

<dependencies>
<!-- Other Dependencies -->
    <dependency>
        <groupId>com.github.unsupervise</groupId>
        <artifactId>spark-tss_2.11</artifactId>
        <version>0.1.1</version>
    </dependency>
</dependencies>
<repositories>
<!-- Other Repositories ... -->
    <repository>
        <id>bintrayunsupervisemaven</id>
        <name>bintray-unsupervise-maven</name>
        <url>https://dl.bintray.com/unsupervise/maven/</url>
        <layout>default</layout>
    </repository>
</repositories>

score 2 · Answer 1 · answered Apr 29 '20 at 15:44

2

Use the repositories and packages parameters.

spark-shell \
  --repositories "https://dl.bintray.com/unsupervise/maven" \
  --packages "com.github.unsupervise:spark-tss_2.11:0.1.1"

answered Apr 29 '20 at 15:44

Jasper-M

14,966
2
26
37

Aravind Yarram · Answer 2 · 2020-04-29T23:03:06.197

1

When you have jars locally:

./spark-shell --jars pathOfjarsWithCommaSeprated

When the binary artifacts (jars) are managed through an artifact repository like Maven or Nexus then you would use artifact coordinates like group id, artifact id and version.

Reference: http//spark.apache.org/docs/latest/rdd-programming-guide.html#using-the-shell

./bin/spark-shell --master local[4] --packages "com.github.unsupervise:spark-tss_2.11:0.1.1"

edited Apr 29 '20 at 23:03

answered Apr 29 '20 at 15:07

Aravind Yarram

78,777
46
231
327

I don't have the jar, I only have these two instructors in maven or sbt. – zekri sidi mohamed hicham Apr 29 '20 at 15:16
Updated my answer – Aravind Yarram Apr 29 '20 at 15:40

How to add library to spark shell

2 Answers2