Livy POST batche API with spark.driver.extraJavaOptions

Question

How I can add spark.driver.extraJavaOptions with Livy POST/Batch API call? I need to pass additional -D (JVM system properties).

Answer 1

You can refer Livy REST Api doc: https://livy.incubator.apache.org/docs/latest/rest-api.html

When you POST to /bathes the JSON body you can pass "conf": {"key": "value"} map containing the required SparkConf configs. Eg.:

{
  "name": "SparkPi-01",
  "className": "org.apache.spark.examples.SparkPi",
  "numExecutors": 2,
  "file": "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.3.jar",
  "args": ["10000"],
  "conf": {
    "spark.driver.extraJavaOptions": "-D... -XX...",
    "spark.executor.extraJavaOptions": "-D... -XX...",
  }
}

Note that, depending on your environment, you may overwrite the preconfigured default values for the extraJavaOptions configs.