EntLib 5 Caching - suspected deadlock - cpu hangs at 100%

Question

I am having this issue for months now. I upgraded from entlib 4.1 to 5. My application is caching increasingly more items. At times (sometimes trice a day) the CPU hangs at 100% usage but the application stays responsive. I used dotTrace to get a snapshot when this occurs and it appears that most of the time is spent in PriorityDateComparer.Compare. This Comparer is used only by the constructor of the System.Collections.SortedList and contains this body:

public int Compare(object x, object y)
{
   CacheItem leftCacheItem = (CacheItem)unsortedItems[(string)x];
   CacheItem rightCacheItem = (CacheItem)unsortedItems[(string)y];

   lock (rightCacheItem)
   {
         lock (leftCacheItem)
         {
            if (rightCacheItem == null && leftCacheItem == null)
            {
               return 0;
            }
            if (leftCacheItem == null)
            {
               return -1;
            }
            if (rightCacheItem == null)
            {
               return 1;
            }

            return leftCacheItem.ScavengingPriority == rightCacheItem.ScavengingPriority
               ? leftCacheItem.LastAccessedTime.CompareTo(rightCacheItem.LastAccessedTime)
               : leftCacheItem.ScavengingPriority - rightCacheItem.ScavengingPriority;
         }
   }
}

Question 1: Can we be sure that the two cache items are always locked in the same order? I don't think so, if I examine the implementation of SortedList.

Question 2: If the answer to my first question is no then how do we solve this? I see some possibilities:

Remove locking and make sure only one thread is used.
Place one lock on the unsortedItems collection and not on the cacheItems.
Somehow figure out in which order to lock the items, for instance by comparing (string)x and (string)y first, and then lock them in the proper order.
other: ...

What do you prefer?

Ok, this is not a deadlock. Looking closer in the trace: there is only one thread calling the comparer. Looking in the code: before the comparer is used, a lock is placed in the cachemanager. I am convinced this code is not concurrently executed. Looking again at the trace: most time in Compare is taken up by System.Collections.HashTable.get_Item(o). So this comparer is used to sort a HashTable. Not at all best practice: [is it possible to sort a hashtable](http://stackoverflow.com/questions/675759/is-it-possible-to-sort-a-hashtable). — Jeroen Visscher, Jun 25 '12 at 12:12

score 0 · Answer 1 · answered Jul 03 '12 at 20:08

I changed the comparer so that it does not need to lookup the cache items:

  int IComparer<CacheItem>.Compare(CacheItem leftCacheItem, CacheItem rightCacheItem)
  {
     lock (rightCacheItem)
     {
        lock (leftCacheItem)
        {
           if (rightCacheItem == null && leftCacheItem == null)
           {
              return 0;
           }
           if (leftCacheItem == null)
           {
              return -1;
           }
           if (rightCacheItem == null)
           {
              return 1;
           }

           return leftCacheItem.ScavengingPriority == rightCacheItem.ScavengingPriority
               ? leftCacheItem.LastAccessedTime.CompareTo(rightCacheItem.LastAccessedTime)
               : leftCacheItem.ScavengingPriority - rightCacheItem.ScavengingPriority;
        }
     }
  }

And in Microsoft.Practices.EnterpriseLibrary.Caching.ScavengerTask I changed the calling method accordingly from:

  private static SortedList SortItemsForScavenging(Hashtable unsortedItemsInCache)
  {
     return new SortedList(unsortedItemsInCache, new PriorityDateComparer(unsortedItemsInCache));
  }

to

  private static List<CacheItem> SortItemsForScavenging(Hashtable unsortedItemsInCache)
  {
     List<CacheItem> cacheValues = new List<CacheItem>(unsortedItemsInCache.Values.Cast<CacheItem>());
     cacheValues.Sort(new PriorityDateComparer());
     return cacheValues;
  }

EntLib 5 Caching - suspected deadlock - cpu hangs at 100%

1 Answers1