2

We have a web application written in MVC2 + Linq2SQL + MS SQL SERVER 2008 hosted on the web farm. Almost like stackoverflow.com

We have 4 x IIS7 + 1 SQL SERVER 2008 load balanced with MS NLB

Static content is cached by external cache provider - Akamai, which reduces 86% of the requests.

Each web server has 32 GB of RAM and 4 x quad core CPUs so there are 64 cores on the front-end.

We save a session state in tables.

It works prefectly with medium traffic (page load = 0.2 s) but the company does the tv ads and during those ads the traffic hits up to 20,000 users within 20-30 seconds.

In this moment the page slows down to 8 - 10 seconds. However, the usage of CPU's and the memory doesn't even arrive to 40% on any machine.

The bandwidth of the data centre does not arrive to the half of its limits.

Pages which are slow generate data from simple SELECTs of maximum 10 records from 1-2 tables only.

Clearly there is a bottleneck somewhere and trying to figure out where.

Anyone has any advice for me where to look for a problem?

Artur Kedzior
  • 3,994
  • 1
  • 36
  • 58
  • 1
    I guess too many SELECT on those tables is causing this bottelneck. Do you have proper indexes etc on those tables based on your WHERE clause? – Ankur Oct 04 '11 at 07:13
  • I agree with Ankur. Sounds like an excessive amount of queries is executed on the database. Check your indexes, usage of ToList(), Any() etc. Those are all functions that make calls to the database. You could also use a tool like DotTrace to profile your application and see what happens. – Rob Oct 04 '11 at 07:16
  • I have clustered indexes on all PK columns and I have gone through the execution plans. I don't think that lack of additional indexes would cause this bottleneck. It certainly helps but it would be a small percentage that I would gain. I'm looking for a big gain. Therefore I was also thinking about Disk I/O of the SQL server box and separating the database into several different files placed on the several different disks to improve the concurrency of the Disk I/O. As it says here: http://tinyurl.com/6xfxuso – Artur Kedzior Oct 04 '11 at 10:16

2 Answers2

1

I found the problem sometime ago but didn't have a time to post it here. I have tried almost every possible thing I could think of: indexes, optimizing sql, monitoring disk i/o, revising the code. Nothing of the above solved my problem. I run the web stress test from 2 workstations on our lan and I could freeze the whole page in seconds! While running tests I have noticed something odd. I created a separate controller for testing and I put 2 methods. Both doing the same thing: asking for the same data our database server.

Action A)

public ActionResult Index(){
var model = new SomeModel();
// Get data
....
return View(model);
}

Action B)

public ActionResult Index() {
 var model = new SomeModel();
// Get data
....
return View("Index",model);
}

The difference in page load:

A) ~ 297 ms

B) ~ 39.47 s

This occurs only during the high concurrency, otherwise is fine. Seems like the search of default View in MVC2 library is causing the problem as once it is specified it all works great.

I have described it on my blog:

http://arturito.net/2011/10/24/asp-net-mvc2-in-the-web-farm-slow-page-load-with-high-traffic-where-is-the-bottleneck/

I have posted it in Issue Tracker of asp.net on codeplex but nobody has commented/responded to it yet.

http://aspnet.codeplex.com/workitem/9396

Artur Kedzior
  • 3,994
  • 1
  • 36
  • 58
0

For some smaller performance benefits, you may try some of the tweaks Rico Mariani mentions at http://blogs.msdn.com/b/ricom/archive/2007/06/22/dlinq-linq-to-sql-performance-part-1.aspx. Specifically try using CompiledQuery and turning off change tracking for display only objects.

Jim Wooley
  • 10,169
  • 1
  • 25
  • 43