2

First of all, I apologize if I use the wrong terminology here, I'm not particularly familiar with the inner workings of caching or Linux file systems. For that matter, I may be misunderstanding the problem and barking up the wrong tree.

Currently running grails 3.2.10 with

compile 'org.grails.plugins:cache-ehcache:3.0.0.M1'

This seems to use ehcache 3.1.4 under the covers. We're having a problem where our disk-based caches are being loaded on startup that are leaving too many "open files". With our applications keeping over 4000 open files for .jars, the 900+ open files for reading cache are causing our server to report "Too Many Open Files" and the server becomes unusable. We've increased the available open files, but it looks like we shouldn't have nearly this many files open for caches.

Sample cache config:

<config xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='http://www.ehcache.org/v3' xsi:schemaLocation="http://www.ehcache.org/v3 http://www.ehcache.org/schema/ehcache-core-3.1.xsd">
<persistence directory='/var/folders/x3/gs0rdtkn79s_5gc98d_0d4w0002yjd/T/myApp/ehcache'/>

<cache-template name="default">
    <key-type>java.lang.Object</key-type>
    <value-type>java.lang.Object</value-type>
    <heap unit="entries">1000</heap>
</cache-template>


<cache alias='myCacheName' uses-template='default'>
<key-type>java.lang.String</key-type>
<value-type>java.util.ArrayList</value-type>
<expiry><ttl unit='hours'>2</ttl></expiry>
<resources>
  <heap unit='entries'>1</heap>
  <disk unit='MB' persistent='true'>50</disk></resources>
</cache>
</config>

When running my app, if I check open files using lsof -p <pid>, I see that the following file is opened 34 separate times, and this is kept open after the application loads:

/private/var/folders/x3/gs0rdtkn79s_5gc98d_0d4w0002yjd/T/myApp/ehcache/myCacheName_863f4553eb5da54a98fff5c3f33a6a863225f6b8/offheap-disk-store/ehcache-disk-store.data

This pattern repeats for every disk-based cache. This is the same when running locally and when deploying .wars to a Tomcat server. I've tried tracing through the ehcache code to find where these are opened, but so far unsuccessfully, ehcache is quite wily.

Does anyone know why these disk-based caches are opened so many times (and not closed)? Is there a way to configure this so that the caches are "warmed" differently?

Edit, more information:

So I've finally figured out what's happening, though not yet how to fix it. ehcache is using a terracotta implementation for disk caches (I'm not sure if this is the default implementation, or the only included implementation) which focuses on performance. For disk-based caches, the default concurrency is hard-coded to 16, which means it opens 16 read and 16 write FileChannels to each disk cache (and then two other open file handles for some reason, but not that important at this point). Neither from combing through the code, nor from the documentation have I found a way to configure this.

Trebla
  • 1,164
  • 1
  • 13
  • 28
  • Have you tried increasing from 1000 to 10000 to see what a difference it would make and possibly increasing megs. Why cache to disk I try to use memory then fall over to disk if its a must but there maybe a genuine reason – V H Aug 22 '17 at 20:10
  • We have increased the open file limit to prevent the server from crashing, we've also converted some disk-based caches to in-memory, but some "need" to remain on disk (primarily due to size). We're working around the issue, but also trying to resolve the underlying problem so we don't hit the same problem as we grow. – Trebla Aug 23 '17 at 11:39

0 Answers0