0

Exception Message: Timeout performing GET allBots, inst: 1, mgr: Inactive, err: never, queue: 7, qu: 0, qs: 7, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0, IOCP: (Busy=2,Free=998,Min=1,Max=1000), WORKER: (Busy=0,Free=2047,Min=1,Max=2047)

Timeout performing GET stock_by_symbol_leg.to, inst: 1, mgr: Inactive, err: never, queue: 13, qu: 0, qs: 13, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, IOCP: (Busy=3,Free=997,Min=1,Max=1000), WORKER: (Busy=3,Free=2044,Min=1,Max=2047)

Timeout performing GET stock_by_symbol_aapl, inst: 1, mgr: Inactive, err: never, queue: 13, qu: 0, qs: 13, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, IOCP: (Busy=3,Free=997,Min=1,Max=1000), WORKER: (Busy=3,Free=2044,Min=1,Max=2047)

Timeout performing GET portefoliosBotById_ec030000-0001-1200-0000-000000000000, inst: 1, mgr: Inactive, err: never, queue: 13, qu: 0, qs: 13, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, IOCP: (Busy=3,Free=997,Min=1,Max=1000), WORKER: (Busy=3,Free=2044,Min=1,Max=2047)

I am using StackExchange.Redis version 1.1.603 and the smallest Azure instance available. I am getting a lot of GET/SET timeout error. I do not have this issue when working locally with a Redis server on my box which make me lean that the problem in on Azure. The information stored in Redis is about 2kb to 10kb.

On Azure portal, I see my number of connection under 20, memory usage around 130meg, cpu usage under 35%, and the Redis server load is always below 13%. I do not see any indication of problem from the portal.

Any idea where I can get more information to work that problem out?

Edit

Since my initial post, I improved few settings.

1) I passed from C0 to C1 instance of Azure Redis.

2) I changed the connection string to have 15 sec timeout. Here is how it looks:

boursexxxxxxx.windows.net:6380,password=xxxxxxxx,ssl=True,abortConnect=False,connectRetry=5, connectTimeout=15000, synctimeout=15000"

3) I created a pool of 10 Lazy Loading ConnectionMultiplex that I am rotating on each call.

private static readonly Lazy<ConnectionMultiplexer>[] lazyConnection;

//In the static constructor of my cache :

lock (lockPookRoundRobin)
{
                lazyConnection = new Lazy<ConnectionMultiplexer>[POOL_SIZE];
                var connectionStringCache = System.Configuration.ConfigurationManager.AppSettings["CacheConnectionString"];
                for (int i = 0; i < POOL_SIZE; i++)
                {
                    lazyConnection[i] = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(connectionStringCache));
                }
}

4) I reduced the value of many cache. Depending of the object serialized I have :

4.1) 1.25ko (5%)

4.2) 0.154ko (5%)

4.3) 26ko (20%)

4.4) 700ko (here we need to work on, but it's limited to < 100 entries)

4.5) 5ko (30%)

4.6) 66ko (40%)

Patrick Desjardins
  • 136,852
  • 88
  • 292
  • 341

2 Answers2

2

Looks like your ThreadPool settings need to be adjusted as described here.

You may also want to review these articles as well as they will help you be aware of common issues that people run into when using Redis:

JonCole
  • 2,902
  • 1
  • 17
  • 19
  • Nice source of information. I have most of them. I'll increase Azure from the Basic to a standard size (even if it's very expensive). I'll try to have a pool of ConnectionMultiplexer. I'll write back my result soon. – Patrick Desjardins Aug 02 '16 at 04:22
0

For everyone that are reading that post. I did few things:

  • If you are using Entity Framework, be sure to limit the number of deepness you go during serialization.
  • Compress the serialized string.
  • Use C1 instead of C0.
  • Add a lot of telemetry to find the frequency of setting and getting from the cache, optimize heavy scenario.
  • Cache the Redis cache with a Memory Cache on IIS. I did something very small, but was releasing the pressure for case with heavy data need.

Being on the cloud, with Redis not on the same server as the webservice add a significant overhead.

Patrick Desjardins
  • 136,852
  • 88
  • 292
  • 341