Store multiple byte-arrays in native memory

Question

I have a fixed amount of byte-arrays (byte[]) of fixed length that I want to store in native memory (and later retrieve). However, I'm not too sure how I can directly store multiple arrays in a MemorySegment.

I know that I could potentially create one large MemorySegment and initialize it element by element, but I suppose that this strategy would be slow and would make retrieval more troublesome (maybe?).

In the API-documentation I came across a SegmentAllocator abstraction, which seems to solve my problem for the allocation, but I don't understand how to retrieve the allocated data using this SegmentAllocator.

try(ResourceScope scope = ResourceScope.newConfinedScope()){
    byte[] data = objToByteArray(someClass.getDeclaredConstructor().newInstance());  //suppose data is always of constant length                    
    SegmentAllocator alloc = SegmentAllocator.arenaAllocator(numOfArrays* data.length, scope);
    for(int i = 0; i < numOfArrays; i++){
        alloc.allocateArray(MemoryLayouts.JAVA_BYTE, data);
        data = objToByteArray(someClass.getDeclaredConstructor().newInstance());
    }
    //how can I access the arrays in alloc?
}catch(Exception e){
    e.printStackTrace();
}

Is there a way to access the data in the SegmentAllocator or is there maybe a different approach to solve my problem?

You seem to be ignoring the return value of `allocateArray`, which returns the `MemorySegment` containing the data. — Kayaman, Jun 15 '21 at 19:28

score 0 · Accepted Answer · answered Jun 19 '21 at 13:41

It seems that what you want to do here is copy these byte arrays end-to-end into some memory segment.

You should start by first allocating a big memory segment, slicing it up in a loop, and then copy a byte array into a slice at a time:

try (ResourceScope scope = ResourceScope.newConfinedScope()) {
    byte[] data = objToByteArray(someClass.getDeclaredConstructor().newInstance());  //suppose data is always of constant length
    long stride = data.length;
    // allocate segment
    MemorySegment segment = MemorySegment.allocateNative(numOfArrays * stride, scope);
    
    // copy arrays to segment
    for (int i = 0; i < numOfArrays; i++) {
        MemorySegment slice = segment.asSlice(i * stride, stride);
        slice.copyFrom(MemorySegment.ofArray(data));
        data = objToByteArray(someClass.getDeclaredConstructor().newInstance());
    }

    // read arrays back
    for (int i = 0; i < numOfArrays; i++) {
        MemorySegment slice = segment.asSlice(i * stride, stride);
        data = slice.toByteArray();
        System.out.println(Arrays.toString(data));
    }
} catch (Exception e) {
    e.printStackTrace();
}

(also, FWIW, you end up creating an extra byte array on the last iteration of the fill loop that is never actually used).

Thank you for your answer. I actually ended up finding a very similar approach. However I am not sure if this strategy is faster than manually copying the byte arrays into the segment (element by element) (since I have multiple threads constantly writing data into the segment (concurrently)). I believe the continuous creation of MemorySegments (-->asSlice) would have a negative impact on the performance (Garbage Collection overhead). — sqlearner, Jun 19 '21 at 17:56

Store multiple byte-arrays in native memory

1 Answers1