5

I'm working on Jupyter Notebooks using sparkmagic kernel (spark-scala) which relies on Apache Livy to run spark jobs.

I'm currently trying to understand the options to create sessions with user-provided dependencies i.e., jars.

I know in Jupyter I can do something like:

%%configure -f
{"kind": "spark", 
 "executorMemory": "2G", 
 "executorCores": 4, 
 "jars": ["/path/to/library1.jar",
         "/path/to/library2"]}

But then I noticed other options in LIVY_HOME/conf/livy.conf, particularly livy.resc.jars and livy.repl.jars but I believe the descriptions are not very specific or I'm unable to find the actual difference between these.

I guess if they are separated it's for something, but what exactly?

Ohtar10
  • 128
  • 1
  • 11

0 Answers0