2

I need to put a properties file that my spark application uses in spark driver classpath.

As per the documentation, it looks like --driver-class-path should do this but it didn't work for me. I tried following. (lets say /home/myuser/ is the location of my property file)

  1. --driver-class-path /home/myuser/
  2. --driver-class-path /home/myuser/*
  3. --driver-class-path /home/myuser/,/home/myuser/*
  4. --driver-class-path /home/myuser/application.properties>

None of them worked.

I tried the same with --conf "spark.driver.extraClassPath=...." as well with no luck.

Any help is highly appreciated.

Bijith Kumar
  • 181
  • 7
  • not sure I understand the problem but why not just pass this path as an argument to the spark application and parse it in main method? – Tal Joffe Apr 04 '17 at 13:24
  • 1
    @TalJoffe The problem is that --driver-class-path doesn't seem to work as I expected. Agree that passing the path as application argument is a work around. However, it is just a work around. Ideally, we should be able to add paths to the classpath and not pass it every time the application is submitted. Another way to do it is to execute spark-submit from the property file folder. Again not as clean as adding classpath – Bijith Kumar Apr 04 '17 at 17:58
  • I too have a similar requirement. We add --files to pass more files and refer them in code with just their filename. This is working great with yarn and cluster mode. But sometimes when I have issue with yarn queue as many users share the queue in our dev environment. I would like to use local and client mode. Tried whatever you have tried. But no luck yet. Wonder how others do and what is best practice. I know this is old but the context is built here. So waking old question. – manu Mar 08 '23 at 16:51

0 Answers0