Why doesn't larger Xss give bigger maximum stack depth?

Question

Firstly, Code:

public class StackSOF {
    private int deep = 0;
    public void stackLeak() {
        deep++;
        stackLeak();
    }    
    public static void main(String[] args) {
        StackSOF sof = new StackSOF();        
        try {
            sof.stackLeak();
        } catch (Throwable e) {
            System.out.println(" stack deep = " + sof.deep + "\n\r" + e);
        }
    }
}

On macOS 10.15.2, with 11.0.5+10 from AdoptOpenJDK.

$ java -Xint -Xss159k StackSOF
 stack deep = 4422
java.lang.StackOverflowError
$ java -Xint -Xss160k StackSOF
 stack deep = 668
java.lang.StackOverflowError
$ java -Xint -Xss161k StackSOF
 stack deep = 4422
java.lang.StackOverflowError

Why is the max stack depth of 160k less than 159k?

Answer 1

I believe this is incorrect behavior that needs to be fixed. I've submitted the bug JDK-8236569.

It turns out that -Xss not multiple of 4K does not work on macOS at all.

-Xss argument is processed twice. The stack size of the very first (Main) thread is configured by the launcher. The launcher simply calls pthread_attr_setstacksize without preprocessing the argument.

However, the Mac OS X documentation to pthread_attr_setstacksize explicitly states that the function returns EINVAL, if stacksize is not a multiple of the system page size. This behavior differs from Linux, where pthread_attr_setstacksize may accept odd values.

-Xss is also processed by the JVM, and in this case the value is rounded to the page size:

  // Make the stack size a multiple of the page size so that
  // the yellow/red zones can be guarded.
  JavaThread::set_stack_size_at_create(align_up(stack_size_in_bytes, vm_page_size()));

So, new threads will have correctly adjusted stack size. If you change your test code to run in a new thread, it will behave as expected.

public class StackSOF {
    private int deep = 0;
    public void stackLeak() {
        deep++;
        stackLeak();
    }    
    public static void main(String[] args) {
        new Thread(() -> {
            StackSOF sof = new StackSOF();
            try {
                sof.stackLeak();
            } catch (Throwable e) {
                System.out.println(" stack deep = " + sof.deep + "\n\r" + e);
            }
        }).start();
    }
}

Though there will be no difference between 159k and 160k due to rounding:

$ java -Xint -Xss159k StackSOF
 stack deep = 666
java.lang.StackOverflowError
$ java -Xint -Xss160k StackSOF
 stack deep = 666
java.lang.StackOverflowError
$ java -Xint -Xss161k StackSOF
 stack deep = 708
java.lang.StackOverflowError