3

Using spark-submit I'm launching a java program. However I'm setting java arguments for the JVM that are not taken into account. I'm trying to specify the max/min heap free ratio.

However, even though the arguments are present (based on visualvm), the memory graph shows that they don't affect the running program.

Here is the command I'm launching:

"/usr/jdk1.8.0_131/bin/java" "-Xmx4096M" "-Dspark.driver.extraJavaOptions=-XX:MinHeapFreeRatio=30 -XX:MaxHeapFreeRatio=30 -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" "-Dspark.executor.extraJavaOptions=-XX:MinHeapFreeRatio=30 -XX:MaxHeapFreeRatio=30" "-XX:MinHeapFreeRatio=30" "-XX:MaxHeapFreeRatio=30"

As you can see, I've tried setting the arguments on the driver, executor and java command.

Here is the arguments from visualvm: enter image description here

and here is the graph:enter image description here

To be clear, what I'm trying to improve is the "unused" heap size.

When starting the program with these arguments (outside of spark), it behaves correctly.

Majid
  • 654
  • 2
  • 7
  • 28
  • You forgot the **quotes** around `"-Dspark.driver.extraJavaOptions=-XX:MinHeapFreeRatio=30 -XX:MaxHeapFreeRatio=30"` (same with executor). Hence the *Max* options are not forwarded to the driver (nor executor) but applied multiple times to the launcher JVM. Now, if you are running in Local or Client mode, the driver runs inside the launcher so it should inherit the options; but in Cluster mode... – Samson Scharfrichter Jul 28 '17 at 21:38
  • 1
    And _"as you can see"_ it is clearly visible in _"the arguments from visualvm"_. – Samson Scharfrichter Jul 28 '17 at 21:41
  • Actually the quote was in the command, I just missed my copy/paste. I edited question. – Majid Jul 31 '17 at 08:44

0 Answers0