High resident memory on client nodes

Question

We are using Ignite 2.13. We are seeing very high resident memory on client nodes. Below is our configuration,

We are running both client & server nodes with 4Gb heap.
On-heap is not turned on(i.e. default).
No persistent is configured native or rdbms.
The default data region is set with 4Gb max.
Eviction set to LRU and thresholds set to 50%.

However every time the metrics get logged in the log file, on-heap shows some marginal usage in MBs. The off-heap always shows used as 0 for all regions system as well as the default data region we configured in our app.

But when we run ‘top’ the heap is roughly ~3.5Gb while the resident memory/off-heap is at 9Gb and by the time I completed the test it rose to 13 Gb.

I am not sure why is the resident memory increasing so much. And why on the client node? Moreover why do none of the regions show it up in the metric logs? How can I check from ignite tools like visor/etc, what data is getting accumulated in the resident memory?

TIA

score 0 · Answer 1 · answered Feb 24 '23 at 13:24

To enable offheap metrics, they have to be turned on manually:

<bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">
    <property name="dataStorageConfiguration">
        <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
            <property name="metricsEnabled" value="true"/>
            <property name="defaultDataRegionConfiguration">
                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                    <!-- enable mertrics for the default data region -->
                    <property name="metricsEnabled" value="true"/>
                    <!-- other properties -->
                </bean>
            </property>
            <property name="dataRegionConfigurations">
                <list>
                 ....
                </list>
            </property>
        </bean>
    </property>
</bean>

Note, there are 2 <property name="metricsEnabled" value="true"/>, one for DataStorage and another for the default region.

Check other questions like this or this

Thanks @alexandr-shapkin did enable this, it did not show any usage in off-heap. — Victor, Mar 06 '23 at 22:23
You mean it's not showing anything i.e. not working, or the metrics are being printed but doesn't correlate with your issue? — Alexandr Shapkin, Mar 07 '23 at 11:17
Metrics are being published, but there is hardly any usage shown, it's almost 99% free. So further looking into resident memory via jemalloc on what is occupying led to some other objects off another server we use leaking. — Victor, Mar 07 '23 at 18:02

score 0 · Answer 2 · answered Mar 06 '23 at 22:27

Since ignite data region metrics did not show any off-heap usage. I did run jemalloc tool to profile the resident memory to check what is occupying space. It ended up some other service we are using that was leaking objects. Cleaning that up helped get the memory usage down.

High resident memory on client nodes

2 Answers2