what is the better way to iterate over chronicle map of size 3.5 million

Question

I have implemented chronicle map as:

ChronicleMapBuilder
                .of(LongValue.class, ExceptionQueueVOInterface.class)
                .name(MAP_NAME)
                .entries(2_000_000)
                .maxBloatFactor(3)
                .create();

Map has around 3.5 million records. I am using this map to server as cache for search operation. Many times to search for list of entries we have to iterate over whole map. 3.5 Million records taking 8.7 GB of RAM and my java heap allocated memory is 36 GB. Total RAM in server is 49 GB. When i am iterating map using below code:

map.forEachEntry(entry -> {
        if(exportSize.get() != MAX_EXPORT_SIZE && TeamQueueFilter.applyFilter(exceptionFilters, entry.value().get(), queType, uid)){
            ExceptionQueue exceptionQueue = getExceptionQueue(entry.value().get());
            if(exceptionFilters.isRts23Selected() || exceptionQueue.getMessageType().equals(MessageType.RTS23.toString())){
                exceptionQueue.setForsId(Constant.BLANK_STRING);
                rts23CaseIdMap.put(String.valueOf(entry.key().get().getValue()), exceptionQueue);
            }else {
                handleMessage(exceptionQueue, entry.key().get(), caseDetailsList);
            }
            exportSize.incrementAndGet();
        }
    });

it always gives me memory error. VM.array size is less than available memory. Any tips here to iterate over this large map. Thanks for help.

Are you sure that the memory problems are connected with the way you’re iterating and not with the fact that you are populating another map with `rts23CaseIdMap.put(…` within the action, using a newly created `String` as key? — Holger, Jun 29 '20 at 10:11
@Holger, yeah it's failing while reading chronicle-map because i've monitored heap during that time and it was under control. Looks like when we call `map.forEach` it copies all map content into byte array and failing to allocate more memory to byte array. — imnitesh, Jun 29 '20 at 10:22

score 0 · Answer 1 · answered Jun 29 '20 at 12:25

I have to guess as you haven't provided any error here that the error message you are getting says something like java.lang.OutOfMemoryError: Direct buffer memory.

Chronicle Map will not allocate its contents on heap, instead it uses offheap memory-mapping, and likely you're running out of offheap space. You need to check what is the size of the Chronicle Map's file on the disk, the same amount of the offheap memory will be used for mmapping.

Offheap memory can be adjusted using the -XX:MaxDirectMemorySize flag.

what is the better way to iterate over chronicle map of size 3.5 million

1 Answers1