1

The question(s): Am I doing something wrong? Incorrect config setting? Is the amount of memory usage by AppFabric shown below normal?

The problem: I am reading the data from a SQL database table into AppFabric cache memory. It seems like AppFabric uses a large amount of memory for a fairly small object and I cant understand why(I have recently started using ApppFabric-so I am a noob with it)

Description: A SQL table of about 60MB converts to about 800MB once in AppFabric cache.

Details:

SQL data size of table I intend to load:

SQL data size

App Fabric idle memory usage:

AppFabric idle usage

Server Config:

<?xml version="1.0" encoding="utf-8"?>
    <configuration>
        <configSections>
            <section name="dataCache" type="Microsoft.ApplicationServer.Caching.DataCacheSection, Microsoft.ApplicationServer.Caching.Core" />
        </configSections>
         <dataCache size="Small">
<caches>
  <cache consistency="StrongConsistency" name="MobileCache" minSecondaries="0">
    <policy>
      <eviction type="None" />
      <expiration defaultTTL="1" isExpirable="false" />
    </policy>
  </cache>
  <cache consistency="StrongConsistency" name="default" minSecondaries="0">
    <policy>
      <eviction type="Lru" />
      <expiration defaultTTL="10" isExpirable="true" />
    </policy>
  </cache>
</caches>
<hosts>
  <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
    hostId="1073911731" size="1000" leadHost="true" account="BGZA\accName"
    cacheHostName="AppFabricCachingService" name="hostname.domain.co.za"
    cachePort="22233" />
</hosts>
<advancedProperties>
<transportProperties connectionBufferSize="131072" maxBufferPoolSize="2147483647" 
                   maxBufferSize="2147483647" maxOutputDelay="2" channelInitializationTimeout="60000" 
                   receiveTimeout="600000"/>
  <securityProperties>
    <authorization>
      <allow users="Rossp0033" />
    </authorization>
  </securityProperties>
</advancedProperties>
<deploymentSettings>
  <deploymentMode value="RoutingClient" />
</deploymentSettings>

Client Config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="dataCacheClient"
         type="Microsoft.ApplicationServer.Caching.DataCacheClientSection, Microsoft.ApplicationServer.Caching.Core,Version=1.0.0.0,Culture=neutral, PublicKeyToken=31bf3856ad364e35"
          allowLocation="true"
          allowDefinition="Everywhere"/>
  </configSections>
  <dataCacheClient>
    <hosts>
      <host
         name="HostName.DomainName.co.za"
         cachePort="22233"/>
    </hosts>
    <transportProperties connectionBufferSize="131072" maxBufferPoolSize="2147483647"
                      maxBufferSize="2147483647" maxOutputDelay="2" channelInitializationTimeout="60000"
                      receiveTimeout="600000"/>
  </dataCacheClient>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
  </startup> 
</configuration>

The C# code:

public static void ReadPortedNumbers()
{
    MobileDataContext mdc = null;
    List<PortedNumberCollection> col;

    try
    {
        mdc = new MobileDataContext(strConnString);

        col = (from RN in mdc.tblRoutedNumbers
               select Convert(RN)).ToList();

        var CacheFactory = new DataCacheFactory();
        var myCache = CacheFactory.GetCache("MobileCache");

        myCache.Put("PortedNumberCollection", col);

    }
    catch (Exception E)
    {
        throw new System.Exception(E.GetType().ToString() + " in BG.Mobile.DAL.ReadPortedNumbers, Message : " + E.Message, E);
    }
    finally
    {
        if (mdc != null) mdc.Dispose();
    }
}

public static PortedNumberCollection Convert(tblRoutedNumber DataClass)
{
    try
    {
        PortedNumberCollection BusinessClass = new PortedNumberCollection();

        BusinessClass.PortedID = DataClass.PortedID;
        BusinessClass.MSISDN = DataClass.MSISDN;
        BusinessClass.RoutingLabel = DataClass.RoutingLabel;
        BusinessClass.RouteAction = DataClass.RouteAction;

        return BusinessClass;
    }
    catch (Exception E)
    {
        throw new System.Exception(E.GetType().ToString() + " in BG.Bus.Mobile.DALConvertor.Convert(tblRoutedNumber DataClass): " + E.Message);                

    }
}

[DataContract][Serializable]
public class PortedNumberCollection
{
    [DataMember]        
    public Int64 PortedID;

    [DataMember]        
    public string MSISDN;

    [DataMember]        
    public string RoutingLabel;

    [DataMember]
    public string RouteAction;
}   

AppFabric memory usage once data is loaded(put): AppFabric memory usage once data is loaded(put)

BossRoss
  • 869
  • 2
  • 11
  • 31
  • I don't know if it explains it, but most memory caches *over allocate* to prevent lots of allocations. Try loading **half** the table (ideally at random) and see whether that uses half the memory appfabric is currently using, vs it using the exact same amount. – Marc Gravell Nov 21 '13 at 08:21
  • BTW: filling a cache with a large object on a single key (`"PortedNumberCollection"`) is a **really** bad idea; whenever you access it the entire thing will have to be sent, which will hammer the bandwidth. – Marc Gravell Nov 21 '13 at 08:26
  • Additonal thought: try serializing `col` with `NetDataContractSerializer` (which is, IIRC, what the app-fabric client library uses); see how big that is – Marc Gravell Nov 21 '13 at 08:26
  • I understand the over allocation actions by AppFabric but taking your advice and loading half the table gives me the same memory usage as loading the full table(good question that), which leads me to believe something is wrong... – BossRoss Nov 21 '13 at 12:21
  • [Looks like the same problem here](http://social.msdn.microsoft.com/Forums/vstudio/en-US/edfd1958-115f-41d6-a1b1-e67c6c007fe8/app-fabric-service-continually-consuming-memory?forum=velocity) – BossRoss Nov 21 '13 at 12:46
  • You should be loading large tables into memory instead of a distributed network cache. Rethink your application design. – George Filippakos Jul 15 '14 at 15:14

2 Answers2

1

If you're using AppFabric 1.1, you need to know that it's internal GC doesn't release any memory allocated. So if you're certain of the amount of memory you want to use OR have LRU enabled, you can limit the memory for the CacheCluster Host (e.g. SET-CacheHostConfig -HostName ServerName -CachePort PORT -CacheSize 1024). This will ensure your memory usage is not ABOVE that level. It is important to understand that it will creep UP TO that level inevitably.

zaitsman
  • 8,984
  • 6
  • 47
  • 79
  • I did exactly what you mentioned. Unfortunately I did not experience the same results. Nor did your answer answer the question. See the others that did what you mentioned and does not work.[So why the down vote](http://social.msdn.microsoft.com/Forums/vstudio/en-US/edfd1958-115f-41d6-a1b1-e67c6c007fe8/app-fabric-service-continually-consuming-memory?forum=velocity) – BossRoss Jun 03 '14 at 04:19
  • @BossRoss So you're saying that you have 1.1 with all the latest patches, system patched, under 4 GB ram and you can't cap it? I saw that it caps with a bit of overhead (between 50 to 250 Mb on top of cap, but it does cap consistently). What OS are you on? I used this on both 2008 R2 and 2012. – zaitsman Jun 03 '14 at 05:12
-2

I would like to answer my own question for any future users who read this.

Do not use AppFabric rather look at products like Redis or MemCached. They are better in every way that I can see.

AppFabric still has way too many problems for it to be in my production systems.

BossRoss
  • 869
  • 2
  • 11
  • 31
  • 1
    you didn't answer your own question. Your answer is not very relevant either. How much "memory" task manager is showing for appfabric cache service is not related to how much data is stored there (think of SQL server for a good example). Here are a couple of links to help: Best practices: http://msdn.microsoft.com/en-us/library/hh830824(v=azure.10).aspx Somewhat of an Explanation: http://blogs.msdn.com/b/distributedservices/archive/2013/10/02/windows-server-appfabric-memory-consumption-questions.aspx I don't know the exact math, but yes, I can see 10x increase in data in cache. – dawebber Nov 28 '13 at 16:15
  • 1
    I have been threw all of that... several times. The problem is even with Garbage collection, all the correct settings, more than enough memory and CPU using the Put option always increased the memory where it should have replaced the object and the memory (IMO). Unless someone can provide me with a working sample of the same object being "Put" into the same cache area without the memory continuously growing then my answer stands as it is NOT ready for MY production system. – BossRoss Nov 29 '13 at 05:36
  • and just for reference: [link](http://social.msdn.microsoft.com/Forums/vstudio/en-US/edfd1958-115f-41d6-a1b1-e67c6c007fe8/app-fabric-service-continually-consuming-memory?forum=velocity) – BossRoss Dec 02 '13 at 08:37
  • I've been using AppFabric (Velocity CTP3) for over 4 years on a production cluster (yes, I know CTP wasn't supposed to be built for production) with over 1M mobile devices quite successfully. I just recently moved to AF 1.1, so I'm still learning the differences, but the system is just fine. Yes, the memory allocation thing looks scary, but I'm not concerned about it, as long as I don't see timeouts and errors. I constantly update the objects in memory by key. Not sure what you are looking for example-wise. – dawebber Dec 12 '13 at 03:52
  • As one who still uses AppFabric, let me give my whole-hearted support to this answer, especially the bolded part. Furthermore, this answer is more relevant and more correct than ever. – MrBoJangles Apr 23 '18 at 16:01