1

I have a JNA application that needs to compute the sum of elements in a large 1D array (~1G). Using the JNA Resource.Rlimit class I am able to increase the resource limit as shown in the following snippet:

final Resource.Rlimit limit = new Resource.Rlimit();      
limit.rlim_cur = 1024L * 1024L * 1024L;
limit.rlim_max = 2 * 1024L * 1024L * 1024L;

However, this only works when the stack size is set inside the method allocating the large array. Thus, when I run the below code, there is no out of memory exception.

package com.jnaapps;

import com.sun.jna.Library;
import com.sun.jna.Pointer;
import com.sun.jna.Native;
import com.sun.jna.Memory;
import com.sun.jna.platform.unix.LibCAPI;
import com.sun.jna.platform.unix.Resource;

public class Main {
    public interface CLibrary extends LibCAPI, Library {
        CLibrary INSTANCE = (CLibrary) Native.load("c", CLibrary.class);
    }

    public interface NativeClass extends Library {
        NativeClass INSTANCE = (NativeClass)Native.load("nativemethods", NativeClass.class);

        double sumOfArray1d(final Pointer values, long numberOfValues);
    }

    public static void main(String[] args) {
        Test0();
    }


    public static void Test0() {
        final Resource.Rlimit limit = new Resource.Rlimit();
        limit.rlim_cur = 1024L * 1024L * 1024L;
        limit.rlim_max = 2 * 1024L * 1024L * 1024L;

        int err = CLibrary.INSTANCE.setrlimit(Resource.RLIMIT_STACK, limit);
        if (err != 0) {
            int lastError = Native.getLastError();
            System.out.println("An exception occurred while setting RLimit: " +
                    lastError);
        }

        NativeClass clib = NativeClass.INSTANCE;

        long dim = 10000L * 10000L;

        final Pointer bigVector = new Memory(dim*Native.getNativeSize(Double.TYPE));

        for (long i=0; i<dim; ++i) {
            bigVector.setDouble(i*Native.getNativeSize(Double.TYPE), 3.5e-6);
        }

        double total = clib.sumOfArray1d(bigVector, dim);

        System.out.println("Sum of elements in big vector = " + total);
    }

    public static void Test1() {
        NativeClass clib = NativeClass.INSTANCE;

        long dim = 10000L * 10000L;

        final Pointer bigVector = new Memory(dim*Native.getNativeSize(Double.TYPE));

        for (long i=0; i<dim; ++i) {
            bigVector.setDouble(i*Native.getNativeSize(Double.TYPE), 3.5e-6);
        }

        double total = clib.sumOfArray1d(bigVector, dim);

        System.out.println("Sum of elements in big vector = " + total);
    }
}

But when the resource limits are set inside main() and method Test1() is called, the program crashes.

public static void main(String[] args) {
    final Resource.Rlimit limit = new Resource.Rlimit();
    limit.rlim_cur = 1024L * 1024L * 1024L;
    limit.rlim_max = 2 * 1024L * 1024L * 1024L;

    int err = CLibrary.INSTANCE.setrlimit(Resource.RLIMIT_STACK, limit);
    if (err != 0) {
        int lastError = Native.getLastError();
        System.out.println("An exception occurred while setting RLimit: " +
                    lastError);
    }

    Test1();
}

Why can't I increase the stack size in main() before the call to Test1(). The project was written in IntelliJ Idea (Community Version 2019.3) on Ubuntu 16.04 amdx86-64 with 8GB RAM.

Daniel Widdis
  • 8,424
  • 13
  • 41
  • 63
unbound37
  • 109
  • 6
  • This smells like an XY problem: why do you need a *gigabyte* of stack space to sum the elements of a heap-allocated array? – Botje Dec 23 '19 at 08:12
  • @Botje The code presented here is only a minimal reproducible example of a statistical data analysis program written in Java. The current version is very slow because of a large number of 2D arrays (240 x 200) allocated at runtime. My goal is to use JNA to enhance performance of the program. – unbound37 Dec 24 '19 at 17:30
  • Same question though: why is your native code allocating memory on the stack instead of the heap? The stack really is only for your call stack and a few local variables, not gigabytes of data. – Botje Dec 25 '19 at 15:06

0 Answers0