6

I'm trying to determine the cause of a very long (imho) initial start up of an ASP.NET application.

The application uses various third party libraries, and lots of references that I'm sure could be consolidated, however, I'm trying to identify (and apportion blame) the dlls and how much they contribute to the extended startup process.

So far, the start up times vary from 2-5 minutes depending on usage of other things on the box. This is unacceptable in my opinion based on the complexity of the site, and I need to reduce this to something in the region of 30 seconds maximum.

To be clear on the scope of the performance I'm looking for, it's the time from first request to the initial Application_Start method being hit.

So where would I start with getting information on which DLL's are loaded, and how long they take to load so I can try to put a cost/benefit together on which we need to tackle/consolidate.

From an ability perspective, I've been using JetBrains dotTrace for a while, and I'm clear on how benchmark the application once we're in the application, but it appears this is outside of the application code, and therefore outside of what I currently know.

What I'm looking for is methodologies on how to get visibility of what is happening before the first entry point into my code.

Note: I know that I can call the default page on recycle/upgrade to do an initial load, but I'd rather solve the actual problem rather than papering over it.

Note2: the hardware is more than sufficiently scaled and separated in terms of functionality, therefore I'm fairly sure that this isn't the issue.

Martin
  • 2,180
  • 4
  • 21
  • 41
  • 3
    Please decide between "this is unacceptable *for the project*" or "this is unacceptable in *my opinion*", since answers will be dramatically different: one is what (if anything) to do to achieve goals for production application, another - how to entertain/educate yourself about load performance. – Alexei Levenkov Jun 06 '12 at 16:58
  • Startup times of 2-5 minutes are probably going to satisfy both criteria. If the app domain recycles and the service is down for 2-5 minutes, I have trouble envisioning any developer *or* project that would find that acceptable. – Eric J. Jun 06 '12 at 17:01
  • I'll update the question, this is the time before the Application_Start method is hit. – Martin Jun 06 '12 at 19:05
  • Do you use Entity Framework by any chance? View creation of EF can take several minutes. – JulianR Jun 06 '12 at 20:41
  • NHibernate... but I guess it's the same issue... I'm sure this will come up with the processes below. – Martin Jun 06 '12 at 21:40

4 Answers4

5

Separate answer on profiling/debugging start up code:

w3wp is just a process that runs .Net code. So you can use all profiling and debugging tools you would use for normal .Net application.

One tricky point is that w3wp process starts automatically on first request to an application and if your tools do not support attaching to process whenever it start it makes problematic to investigate startup code of your application.

Trick to solve it is to add another application to the same Application Pool. This way you can trigger w3wp creation by navigating to another application, than attach/configure your tools against already running process. When you finally trigger your original application tools will see loading happening in existing w3wp process.

With 2-5 minutes delay you may not even need profiler - simply attach Visual Studio debugger the way suggested above and randomly trigger "break all" several times during load of your site. There is a good chance that slowest portion of the code will be on the stack of one of many threads. Also watch out for debug output - may give you some clues what is going on.

You may also use WinDbg to capture stacks of all threads in similar way (could be more light way than VS).

Alexei Levenkov
  • 98,904
  • 14
  • 127
  • 179
  • I'll attempt a couple of these then I'll mark it as soon as I'm sure it captures the right info. – Martin Jun 06 '12 at 21:40
  • seems that NH is causing the problems... adding the second application to the same appPool allowed me to view the stuff using dotTrace. – Martin Jun 11 '12 at 11:51
2

Your DLL references are loaded as needed, not all at once.

Do external references slow down my ASP.NET application? (VS: Add Reference dialog)

If startup is taking 2-5 minutes, I would look at what happens in Application_Start, and at what the DLLs do once loaded. Are they trying to connect to a remote service that is very slow? Is the machine far too small for what it's doing (e.g. running a DB with large amounts of data plus the web server on an AWS micro instance or similar)?

Since the load time is probably not the IIS worker process resolving references, I would turn to traditional application profilers (e.g. Jetbrains, Antz, dotTrace) to see where the time is being spent as the DLLs initialize, and in your Application_Start method.

Community
  • 1
  • 1
Eric J.
  • 147,927
  • 63
  • 340
  • 553
  • I use the dotTrace to profile the code from the start of Application_Start, but it takes 2-5 minutes to hit that point. Maybe I've not got it setup right to see the dll's initializing? – Martin Jun 06 '12 at 19:22
  • Do you control the source of the (major) DLLs? If so, you could just add a few simple logging statements... Have not used dotTrace in ages unfortunately. – Eric J. Jun 06 '12 at 19:35
  • I do control the site code, unfortunately, the issue is before the site code it seems. Otherwise I would have thought that dotTrace would capture it. – Martin Jun 06 '12 at 22:32
  • 1
    ANTS Performance Profiler can profile FileI/O, so it will show you the time taken accessing files like a .dll, walkthrough is [here](http://www.red-gate.com/supportcenter/Content/ANTS_Performance_Profiler/help/7.2/app_file_io) – Dene B Jun 07 '12 at 07:48
2

Entertainment options check along with profiling:

  • profile everything, add time tracing to everything and log the information
  • if you have many ASPX views that need to be compiled on startup (I think it is default for release configuration) than it will take some time
  • references to Web services or other XML serialization related code will need to compile serialization assemblies if none are present yet
  • access to remote services (including local SQL) may require the services start up too
  • aggressive caching in application/remote services may require per-population of caches

Production:

  • What is the goal for start up time? Figure out it first, otherwise you will not be able to reach it.
  • What is price you are willing to pay to decrease start up time. Adding 1-10 more servers may be cheaper than spending months of development/test time and delaying the product.
  • Consider multiple servers, rolling restarts with warm up calls, web gardens
  • If caching of DB objects or in general is an issue consider existing distributed in-memory caches...
Alexei Levenkov
  • 98,904
  • 14
  • 127
  • 179
  • I like the list, the first has somethings that I've not considered or looked into (Serialization assembilies, compiling ASPX pages). However, what I'm looking for is the ability to get visibility on what cost each of the list is in relation to the whole startup process. – Martin Jun 06 '12 at 19:20
  • I've posted separate answer on tip to debug/profile ASP.Net startup. – Alexei Levenkov Jun 06 '12 at 20:20
0

Despite a large number of dlls I'm almost sure that for a reasonable application it cannot be a cause of problem. Most of the time it is static objects initialization is causing slow startup.

In C# static variables are initialized when a type is first time accessed. I would recommend to use a sql profiler and see what are the queries that are performed during the start time of the application and from there see what are the objects that are expensive to initialized.

Eugeniu Torica
  • 7,484
  • 12
  • 47
  • 62
  • This is a route I've already taken, and there doesn't seem to be that may many, and none which grab huge datasets either. – Martin Jun 06 '12 at 19:23