Can't download packages in spark-shell from password protected private nexus

Question

I am trying to use a private password protected nexus to download and add private jar artifacts to spark-shell classpath but it fails to download it. The spark documentation (https://spark.apache.org/docs/latest/submitting-applications.html) mentions that --repositories argument with uri that looks like https://user:password@host/ should work but my spark-shell still fails to download the artifact. spark-shell prints out a url that it tried and failed but when I click on it I am able to open it in browser just fine.

Here is my example command

./spark-shell --packages com.danish:spark-lib:1.0.0 --repositories https://username:password@nexus.host.com/repository/public

Please let me know if I am missing something. If there are alternate approaches like using mvn or ivy that is connected to nexus, I'd love to hear that as well.

Thanks

Answer 1

I was able to get it to work by using ivy dependency management and managing nexus repository and credentials in ivysetting.xml I created ivysettings.xml file with all nexus the details. see below for example. When running spark shell we point spark.jars.ivySettings to the ivysettings.xml path. for eg

./spark-shell.sh --conf spark.jars.ivySettings=/User/me/ivysettings.xml --packages com.danish:spark-lib:1.0.0

ivysetting.xml example

<ivysettings>
   <settings defaultResolver="nexus" />
   <credentials host="nexus.host.com" realm="Sonatype Nexus Repository Manager" username="username" passwd="mypassword" />
   <property name="nexus-public" value="https://nexus.host.com/repository/public" />
   <property name="nexus-releases" value="https://nexus.host.com/repository/releases" />
   <property name="nexus-snapshots" value="https://nexus.host.com/repository/snapshots" />
   <resolvers>
      <ibiblio name="nexus" m2compatible="true" root="${nexus-public}" />
      <ibiblio name="nexus-snapshots" m2compatible="true" root="${nexus-snapshots}" />
      <ibiblio name="nexus-releases" m2compatible="true" root="${nexus-releases}" />
   </resolvers>
</ivysettings>

This ivysettings.xml has credentials in plain text. I didnt look into encrypting credentials or using a different credentials manager in ivy.