I've set up a production WSO2 API Manager 4.1.0 cluster with two nodes. This cluster is not live yet so it's sitting idle with minimum work to do.
Its memory config is quite generous, especially for the current workload: -Xmx3072m -Xms3072m
The problem I've found is that G1 Old Gen keeps growing until it uses up all the heap memory:
And here's the overall heap view:
As you can see even though there's a huge amount of time for G1 Old Gen (11h+), in the end GC's didn't manage to free much memory until they were pointless.
Eventually the node stars launching Out of memory exceptions:
WSO2 is a widely used API Management solution so the question is, could it have some kind of memory leak?
Is there any way to tweak the G1 Old Gen GC so that it doesn't fill up the whole heap size?
UPDATE:
I've generated a heap dump and the problem seems to be in the org.wso2.carbon.apimgt.gateway component:
Drilling down I reach the org.wso2.carbon.apimgt.gateway.inbound.InboundMessageContextDataHolder:
