1

This question is similar to my other question but this time for Microsoft Azure.

We have a config file that needs to be in the classpath of the driver (and possibly the executors).

When defining a Databricks cluster in Microsoft Azure, I configured custom Spark configs: spark.executor.extraClassPath and spark.driver.extraClassPath. I pointed both paths to a directory in dbfs that contains our config. But as half suspected, this doesn't work.

The question is, which type of paths are supported by these Spark Configs? If it's only local filesystem, how do I get my config file on the driver (and the executors)?

pgruetter
  • 1,184
  • 1
  • 11
  • 29

1 Answers1

0

Please note the setting is case sensitive and I think you should set it to a local directory, and use an init script to copy it there from dbfs (using /dbfs/path syntax), so that it's accessible with low latency from the JVM.

HimanshuSinha
  • 1,650
  • 2
  • 6
  • 10
  • Thanks for the hint, made a mistake when copying the parameter name to StackOverflow, corrected it now. Init Script would work, but this is only run once when the cluster is started, right? I want to be able to change the file and then restart a process. I can't restart the cluster everytime of course. How can I copy the file manually from dbfs to the driver node? – pgruetter Nov 06 '19 at 07:38
  • By the way: Tried to write an init script but failed. Opened another question for that: https://stackoverflow.com/questions/58728321/copy-file-from-dbfs-in-cluster-scoped-init-script – pgruetter Nov 06 '19 at 10:47