Caffeine cache - maximumSize eviction seems not to be late

Question

We have an issue with Caffeine cache. We configured it to be of size 60, with TTL of 300 seconds like this:

Cache<String, String> cache = Caffeine.newBuilder()
                .expireAfterWrite(300, TimeUnit.SECONDS)
                .maximumSize(60)
                .removalListener((String key, String value, RemovalCause cause) -> {
                    cacheListenerHandler(key, value, cause);
                })
                .build();

Now, the removalListener is defined like this:

private void cacheListenerHandler(String key, String value, RemovalCause cause) {
        if (RemovalCause.EXPIRED.equals(cause)) {
            if (value != null) {
                LOG.info("We got TTL expiry of key {} and value {}",
                        key, value);
            } else {
                LOG.warn("Value is null for TTL expiry! key: {}", key);
            }
        }

        if (RemovalCause.SIZE.equals(cause)) {
            if (value != null) {
                LOG.info("We got SIZE expiry of key {} and value {}",
                        key, value);
                //some logic
            } else {
                LOG.warn("Value is null for SIZE expiry! key: {}", key);
            }
        }
    }

With that being said, we insert to the cache this way:

public void registerValue(String key, String value) {
        cache.put(key, value);
        LOG.info("Key {} was added with value {}. Current estimated size of {} keys in cache",
                key, value, cache.estimatedSize());
}

The issue is that sometimes we get logs such as:

Key 'key1' was added with value 'value1'. Current estimated size of 250 keys in cache

And we see constantly the eviction logs (of the listener method):

We got SIZE expiry of key 'key1' and value 'value1'

And a second later the log:

Key 'key2' was added with value 'value2'. Current estimated size of 251 keys in cache

Now, I know about the 'estimatedSize' nuance - it includes the keys that are going to be evicted, but the issue is that we get Java memory heap issues, meaning the actual removal happens too late for use.

Is there a solution for it? Maybe we need to switch to Guava instead?