1

I am having a problem with Ehcache. It usually works fine but occasionally I get a ClosedChannelException when try to access the cache, and once that occurs I can only get round the problem by recreating the cache

Here is my class

package com.jthink.songkong.cache;

    import com.jthink.acoustid.acoustidschema.Result;
    import com.jthink.songkong.analyse.general.Errors;
    import org.ehcache.CachePersistenceException;
    import org.ehcache.config.builders.CacheConfigurationBuilder;
    import org.ehcache.config.builders.ExpiryPolicyBuilder;
    import org.ehcache.config.builders.ResourcePoolsBuilder;
    import org.ehcache.config.units.EntryUnit;
    import org.ehcache.config.units.MemoryUnit;
    import org.ehcache.core.statistics.CacheStatistics;

    import java.time.Duration;
    import java.util.Collection;
    import java.util.HashMap;
    import java.util.Map;

    /**
     * AcoustIdResult EhCache
     *
     */
    public class AcoustIdResultCache
    {
        protected org.ehcache.Cache<String, Result>           cache;

        public static final String ACOUSTID_RESULT_CACHE = "AcoustidResultCache";

        private static AcoustIdResultCache apc;

        private AcoustIdResultCache()
        {
           createCache();
        }

        static
        {
            apc = new AcoustIdResultCache();
        }

        public static AcoustIdResultCache getInstanceOf()
        {
            if(apc==null)
            {
                apc = new AcoustIdResultCache();
            }
            return apc;
        }


        /**
         * Create the Cache
         */
        private void createCache()
        {
            cache = SongKongCacheManager.getCacheManager().createCache(getCacheName(),
                    CacheConfigurationBuilder.newCacheConfigurationBuilder(
                            String.class,
                            Result.class,
                            ResourcePoolsBuilder.newResourcePoolsBuilder()
                                    .heap(300, EntryUnit.ENTRIES)
                                    .disk(500, MemoryUnit.MB, true)
                    )
                            .withExpiry(ExpiryPolicyBuilder.timeToIdleExpiration(Duration.ofDays(10)) //Since Acoustid is Live Db
                            )
            );
        }

        protected AcoustIdResultCache(String cacheName)
        {
            createCache();
        }

        public void init()
        {

        }

        /**
         * Close Cache Manager
         */
        public static void close()
        {
            SongKongCacheManager.getCacheManager().close();
        }

        public void emptyCache()
        {
            try
            {
                SongKongCacheManager.getCacheManager().destroyCache(getCacheName());
                createCache();
            }
            catch(CachePersistenceException cpe)
            {
                Errors.addError("Problem emptying cache for"+getCacheName()+":"+cpe.getMessage(), cpe);
            }
        }

        public void clearCacheStatistics()
        {
            SongKongCacheManager.getStatistics().getCacheStatistics(getCacheName()).clear();
        }

        /**
         *
         * @return cache name
         */
        protected String getCacheName()
        {
            return ACOUSTID_RESULT_CACHE;
        }

        /**
         *
         * @return cache
         */

        protected org.ehcache.Cache<String, Result> getCache()
        {
            return cache;
        }


        /**
         *
         * @return statistics of cache
         */
        public String statusCache()
        {
            CacheStatistics ehCacheStat = SongKongCacheManager.getStatistics().getCacheStatistics(getCacheName());
            return "CacheName:" + getCacheName()
                    + ":Put:"   + ehCacheStat.getCachePuts()
                    + ":Hit:"   + ehCacheStat.getCacheHits()
                    + ":Miss:"  + ehCacheStat.getCacheMisses()
                    + ":Evictions:"  + ehCacheStat.getCacheEvictions()
                    + ":Removals:"  + ehCacheStat.getCacheRemovals();
        }

        /**
         * Add an item to cache
         *
         * @param item
         */
        public void add(Result item)
        {
            addToCache(item);
        }

        /**
         * Get Item from cache
         *
         * @param id
         * @return the item or null
         */
        public  Result get(String id)
        {
            return getFromCache(id);
        }

        /**
         * Is there an item with this id in cache
         * @param id
         * @return
         */
        public boolean isInCache(String id)
        {
            return getCache().containsKey(id);
        }

        /**
         *
         * @param id
         * @return item for this id or null if doesn't not exist
         */
        protected   Result getFromCache(String id)
        {
            return getCache().get(id);
        }

        /**
         * For adding multiple items to cache
         *
         * @param items
         * @return
         */
        public  boolean addToCache(Collection<Result> items)
        {
            try
            {
                Map<String, Result> map = new HashMap<String, Result>();
                for(Result acoustidResult:items)
                {
                    map.put(acoustidResult.getId(), acoustidResult);
                }
                getCache().putAll(map);
                return true;
            }
            catch(Exception e)
            {
                Errors.addError("Failed AddResultToCache:" + e.getMessage(), e);
                return false;
            }
        }

        /**
         * Add item to cache
         *
         * @param item
         */
        protected  boolean addToCache(Result item)
        {
            try
            {
                getCache().putIfAbsent(item.getId(), item);
                return true;
            }
            catch(Exception e)
            {
                Errors.addError("Failed AddAcoustidToCache:"+item.getId()+ ':' +e.getMessage(),e);
                return false;
            }
        }
    }

and this is the stack trace

java.lang.RuntimeException: java.nio.channels.ClosedChannelException
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine$FileChunk$1.setLong(FileBackedStorageEngine.java:677)
    at org.ehcache.impl.internal.store.offheap.LazyOffHeapValueHolder.writeBack(LazyOffHeapValueHolder.java:82)
    at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.lambda$flush$16(AbstractOffHeapStore.java:865)
    at org.ehcache.impl.internal.store.disk.EhcachePersistentConcurrentOffHeapClockCache.lambda$computeIfPinned$3(EhcachePersistentConcurrentOffHeapClockCache.java:213)
    at org.terracotta.offheapstore.OffHeapHashMap.computeIfPresentWithMetadata(OffHeapHashMap.java:2096)
    at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeIfPresentWithMetadata(AbstractLockedOffHeapHashMap.java:604)
    at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeIfPresentWithMetadata(AbstractConcurrentOffHeapMap.java:781)
    at org.ehcache.impl.internal.store.disk.EhcachePersistentConcurrentOffHeapClockCache.computeIfPinned(EhcachePersistentConcurrentOffHeapClockCache.java:210)
    at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.flush(AbstractOffHeapStore.java:858)
    at org.ehcache.impl.internal.store.heap.OnHeapStore.lambda$setInvalidationListener$19(OnHeapStore.java:914)
    at org.ehcache.impl.internal.store.heap.OnHeapStore.lambda$evict$28(OnHeapStore.java:1552)
    at org.ehcache.impl.internal.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1848)
    at org.ehcache.impl.internal.store.heap.SimpleBackend.computeIfPresent(SimpleBackend.java:130)
    at org.ehcache.impl.internal.store.heap.OnHeapStore.evict(OnHeapStore.java:1547)
    at org.ehcache.impl.internal.store.heap.OnHeapStore.enforceCapacity(OnHeapStore.java:1514)
    at org.ehcache.impl.internal.store.heap.OnHeapStore.resolveFault(OnHeapStore.java:745)
    at org.ehcache.impl.internal.store.heap.OnHeapStore.getOrComputeIfAbsent(OnHeapStore.java:692)
    at org.ehcache.impl.internal.store.tiering.TieredStore.get(TieredStore.java:88)
    at org.ehcache.core.Ehcache.doGet(Ehcache.java:90)
    at org.ehcache.core.EhcacheBase.get(EhcacheBase.java:127)
    at com.jthink.songkong.cache.AcoustIdResultCache.getFromCache(AcoustIdResultCache.java:177)
    at com.jthink.songkong.cache.AcoustIdResultCache.get(AcoustIdResultCache.java:157)
    at com.jthink.songkong.analyse.acoustid.AcoustidHelper.getMusicBrainzAcoustidResultsFromCacheOrDb(AcoustidHelper.java:136)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSingleSongMatcher.getReleasesByAcoustid(MusicBrainzSingleSongMatcher.java:99)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSingleSongMatcher.queryForCandidateReleasesByArtistTitleAndReleaseForSingleSong(MusicBrainzSingleSongMatcher.java:335)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSingleSongMatcher.matchSingleSongToReleaseUsingExistingFileMetadata(MusicBrainzSingleSongMatcher.java:396)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSingleSongMatcher.match(MusicBrainzSingleSongMatcher.java:72)
    at com.jthink.songkong.analyse.analyser.AbstractMusicBrainzGroupMatcher.matchSongs(AbstractMusicBrainzGroupMatcher.java:146)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher1.doTask(MusicBrainzSongGroupMatcher1.java:420)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher.call(MusicBrainzSongGroupMatcher.java:469)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher1.call(MusicBrainzSongGroupMatcher1.java:97)
    at com.jthink.songkong.analyse.analyser.MusicBrainzSongGroupMatcher1.call(MusicBrainzSongGroupMatcher1.java:40)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
    at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:758)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.writeBufferToChannel(FileBackedStorageEngine.java:393)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.writeLongToChannel(FileBackedStorageEngine.java:388)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine.access$700(FileBackedStorageEngine.java:60)
    at org.terracotta.offheapstore.disk.storage.FileBackedStorageEngine$FileChunk$1.setLong(FileBackedStorageEngine.java:675)
    ... 35 more

I don't see why happens, my first thought is I should modify

  protected org.ehcache.Cache<String, Result> getCache()
  {
     return cache;
  }

to

  protected org.ehcache.Cache<String, Result> getCache()
  {
      if(cache==null)
      {
          cache = createCache();
      }
      return cache;
  }

but why would cache ever be null ?

My second thought is there is some multithreading issue, but I assume Ehcache is thread safe ?

Paul Taylor
  • 13,411
  • 42
  • 184
  • 351
  • `ClosedChannelException` means that EhCache was unable to write to the file. The channel which was once open is now closed. Are you using some network file storage or something that could become unstable during execution, and cause the open file channel to close? – jbx May 30 '19 at 07:39
  • @jbx Actually the error is just for a particular customer, I dont think they are storing cache on network storage but I am waiting for an answer. – Paul Taylor May 30 '19 at 09:08
  • Well the error is clearly saying that it failed to write to the file because the channel somehow closed. Since this is a FileChannel, something went wrong with the file. – jbx May 30 '19 at 10:36
  • @jbx is it all thread safe, could then be any problem caused by multiple threads reading or writing to cache. – Paul Taylor May 30 '19 at 10:46
  • Well it could be a thread safety issue, but the error doesn't seem to be related to threads so much. Unless somehow one thread closed the channel and another one was expecting it to be still open. You could try to put some explicit locking: https://www.ehcache.org/documentation/2.8/apis/explicitlocking.html – jbx May 30 '19 at 11:00
  • Thats would slow things down somewhat, I used to have this data in a database but then I moved to Ehcache because I realized the data didn't change and so I didn't require full db ACID control. The cache is only accessed via this class so I really wanted to check if my code was okay in a multi thread environment or not. – Paul Taylor May 30 '19 at 13:24
  • Reported github issue: https://github.com/ehcache/ehcache3/issues/2908 – cdalxndr May 27 '21 at 14:37

1 Answers1

1

There have been a couple of bugs in this area. I think that you are falling victim to a variant of: https://github.com/Terracotta-OSS/offheap-store/pull/53

If you could update to Ehcache 3.7.1 which contains the fixes for these issues and see if you can reproduce. If that doesn't solve the issue then please file a bug at https://github.com/ehcache/ehcache3

Chris Dennis
  • 466
  • 2
  • 4