2

Our traffic has gone up significantly, but our site works fine for most of the day, but during our peak hours, our site has been crippled. Requests are being queued in the hundreds and then everything goes back to normal once our peak period is over.

What are the steps to properly diagnose the bottleneck?

We are pretty confident it is not the database server as we use Memcached to cache most of the data. Help is much appreciated.

Thanks!

  • 1
    Do you have any numbers from Performance Monitor (perfmon) during these outtages? Is it cpu/io-bound? If neither, are every thread waiting on something perhaps? Could this be easily solved by changing the asp.net hosting parameters to allow more threads? – sisve Oct 28 '09 at 05:59
  • 1
    There is no way to answer this yet. As Simon says, perfmon numbers for CPU, RAM and disk for starters. Also, are all services running on one server, or how are they divided on servers? How many requests per second? Hardware specs? –  Oct 28 '09 at 22:42
  • Not enough info here... Definitely need number of servers and how these services are divided amongst them. – BatRastered Apr 24 '10 at 00:14

2 Answers2

2

I have added a comment about further information being required to answer this. When we have something more to go on, I'll try to update my answer.

In the meantime, 3 quick things you could try are:

  • Run a profiler on your code. See if your code is doing something horribly slow somewhere.
  • Wrap some timers around key entry and exit points. I.e. store a timestamp when a "key subsystem" is being invoked, and calculate the execution time afterwards. "Key subsystems" are whatever likely causes of slowness you can think of, such as large patches of messy code of your own, 3rd party libraries, memcached, SQL etc. For best results, log to a (ideally separate) database during operation, so that you can follow how time consumption is under peak load.
  • Look at SQL Server; it has good statistics which can tell you a lot. How is the query rate? Are there any queries which a both often used and slow?
1

This is a hard question to answer without more information, like Simon mentioned.

This looks like a good read: http://msdn.microsoft.com/en-us/library/ms998549.aspx

The "Threading Explained" section has some suggested optimizations to reduce request queueing.

user21146
  • 367
  • 1
  • 5
  • 19