How OS handles stack growth of multiple threads with option "ulimit -s unlimited"?

Question

By default, linux stack size is limited to 8 MB. So in case of multi-threaded environment each thread will get its own 8 MB stack. If any thread wanders off the bottom of a stack into the guard page will be rewarded with a segmentation-fault signal. This way we were preventing stacks to overlap with each other or with other memory regions.

However with the help of “# ulimit -s unlimited” we can allocate as much memory possible to stack (till we are not colliding with heap or other memory regions).

My questions are:

After executing “# ulimit -s unlimited”

Where does linux placed stacks of multiple threads in Virtual memory? It cannot be contiguous allocation otherwise they cannot expand.

How it calculate free space between two stacks in virtual memory?, so that they can get equal opportunity to expand.

Answer 1

After executing “# ulimit -s unlimited”

It will invoke syscall setrlimit to setup current shell process's rlimit(see more details from do_prlimit).

Where does linux placed stacks of multiple threads in Virtual memory? It cannot be contiguous allocation otherwise they cannot expand.

Child processes forked from the current bash also inherit its rlimit:

syscall_clone => kernel_clone => copy_process =>
    memcpy(sig->rlim, current->signal->rlim, sizeof sig->rlim);

In Linux, every process has its own view of system memory, you can get it from /proc/pid/maps, which has many regions including stack and each region is represented as a struct vm_area_struct in the kernel, there is no magic. stack is virtual memory continuous, because it is located in a single vm_area_struct for every process, and it never expands or resizes actually. The stack size is determined at exec early time

Let's see how does kernel setup stack each time you call the exec family function:

syscall execv into kernel
=> do_execveat_common
  => call load_binary(for elf, it will invoke `load_elf_binary`)
    => setup_arg_pages(bprm, randomize_stack_top(STACK_TOP),
                 executable_stack);
      => expand_stack(vma, stack_base);
        => expand_downwards(vma, address);
          => anon_vma_interval_tree_pre_update_vma(vma);
          => anon_vma_interval_tree_post_update_vma(vma);