0

I am trying to use logback.xml with Apache Livy ReST API and I'm having trouble getting it to work. I've tried submitting the logback.xml path as follows.

data = {
    "file" : "<PATH_TO_JAR_FILE>",
    "className" : "<CLASS_NAME>",
    "files": ["file:///path/to/log4j.properties", "file:///path/to/logback.xml"],
    "conf": {"driver-java-options":"-Dlogback.configurationFile=file:///path/to/logback.xml"}
}

However, I get the warning Ignoring non-spark config property: driver-java-options=-Dlogback.configurationFile. I can add the logback.xml file using spark.driver.extraJavaOptions instead of driver-java-options, but it doesn't make a difference since the JVM has already started. So my question is, How do I get Livy to use the logback.xml.

rasthiya
  • 650
  • 1
  • 6
  • 20

1 Answers1

0

I see that you're trying to configure logging of Apache Livy server. But using its REST API (IIUC you are sending POST /batches or /sessions) you are submitting Spark job and feeding logback.xml to Spark Driver process, which is separate from Livy Server process.

What you need to do is to pass -Dlogback.configurationFile=... option to Livy Server process on its startup. Most probably (depending on your Livy version and environment) you should set export LIVY_SERVER_JAVA_OPTS=-Dlogback.configurationFile=... environment variable and start Livy with $LIVY_HOME/bin/livy-server start. Refer the script on GitHub.

Aliaksandr Sasnouskikh
  • 921
  • 6
  • 14