I'm working on Jupyter Notebooks using sparkmagic kernel (spark-scala) which relies on Apache Livy to run spark jobs.
I'm currently trying to understand the options to create sessions with user-provided dependencies i.e., jars.
I know in Jupyter I can do something like:
%%configure -f
{"kind": "spark",
"executorMemory": "2G",
"executorCores": 4,
"jars": ["/path/to/library1.jar",
"/path/to/library2"]}
But then I noticed other options in LIVY_HOME/conf/livy.conf
, particularly livy.resc.jars
and livy.repl.jars
but I believe the descriptions are not very specific or I'm unable to find the actual difference between these.
I guess if they are separated it's for something, but what exactly?