1

After perusing stack exchange & numerous MSDN articles for months, I'm coming to the community looking for help. I'm the IIS Admin for my organization & I've been seeing an issue wherein when a change is made to a .NET application, the page takes a very long time to load when a request is first submitted to the page. When I say a long time, I'm saying anywhere from five minutes to a half hour at times. I've tried a number of items to fix the issue. But it's my belief that there's a configuration setting buried somewhere that is causing the issue. Once the page loads once, it's load time is normal going forward. It seems to happen regardless of the application, although some take longer than others. It's also not consistent. Sometimes it'll take seven minutes to load after a change. That same application may take seventeen minutes to load the next time a change is made... The changes are usually very minor, a new image box is moved in, or a new link added. Nothing major.

If an app takes twenty seconds to a minute to load the first time, we're not concerned. But the ten, fifteen minute, sometimes half hour load time is what is not acceptable.

The issue happens regardless of whether or not it's a static content application, or if it's reaching out to a data source. Any connections to a data source are configured at the app level. We only utilize that on a handful of apps & I verified the connection information is valid in the web.config for those apps that do reach out to a data source. We're using Windows Authentication on every app.

We run a three tiered environment, all running Windows Server 2012 R2 Standard with 16gb of ram & a multicore CPU setup. We're running IIS 8.5 on .NET 4.0.30319. The app pool is utilizing the integrated pipeline with support for 32-bit apps (which the apps are). These are VMware Hosts. The server is rebooted once a week. The issue does not occur on our test or dev servers. Only our production server. It doesn't matter what time of day the change is made.

In December of 2016, I ported over all of our .NET applications from an older Windows 2003 system running IIS 6.0 to the new Windows 2012 system. While we worked with the developer to alter any hardcoded hostnames in the apps, we ultimately had to install a CNAME host record to re-direct the old hostname to the new one. The issue seems to have started around this time.

One issue I noticed was that the developer was compiling all apps in debug mode. We changed this setting to false, but it only marginally helped in some cases.

I tried the following as well:

  • Segregated a given app & set the app pool to always run. Also tried changing the application pool identity to run as a network service, or as a service account user.

  • Adding the application initialization role & configuring in IIS to always run the app pool & enabled preload at the app level. - I backed this out when I saw it made no change.

  • I married up all the roles between our test/dev servers where the issue doesn't happen with that of production.

  • Disabled idle timeout at the app pool level.

  • Compared settings between test & production regarding compilation settings, time-outs, etc

None of these changes made any difference. I can't find anything differentiating the test/dev boxes where there is no issue & the production problematic server. Please let me know what additional information you may need & I appreciate the help in advance!!

Thanks, Mike

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
mmostwill
  • 31
  • 1
  • 5

3 Answers3

2

We found the problem. Our AV software was the culprit. Specifically Trend Micro Deep Security.

When a request was made & the program was going to compile in the temp area (C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files) - I noticed it seemed to be taking a very long time for the files to build out. Pre-compiling saw the same behavior. I asked our LAN admin to add an exception for this particular folder. Once that was done, it's only taking 5-10 seconds on first load, which is acceptable. Thanks again for your input.

mmostwill
  • 31
  • 1
  • 5
0

You want to precompile your pages so they don't get compiled on first run. IIRC it's just a simple web.config addition.

https://msdn.microsoft.com/en-us/library/bb398860.aspx

Though to be honest, compile times of 5 minutes per pages seems like there's a bigger problem going on. I've never seen a new aspx compile take longer than a few seconds.

Neil N
  • 24,862
  • 16
  • 85
  • 145
  • "The issue happens regardless of whether or not it's a static content application" also the hour long wait indicates this isn't the problem. Their server is hosed, they need to blow it away and reinstall everything fresh, possibly throw it in the gotdamn trashcan and get a better server. –  Jun 29 '17 at 15:30
  • 1
    @Will I admit I only skimmed his 8 paragraphs. – Neil N Jun 29 '17 at 15:34
  • Sorry for the lengthy post, just wanted to be as thorough as possible. Thanks for the input thus far. All pages run fine after the initial load - I don't know that the server is completely hosed - though if I get the opportunity for a re-build from the higher ups, that option will be taken. – mmostwill Jun 29 '17 at 15:37
0

Neil N's answer makes sense, but I'd definitely check and see if that setting is the same across all of your servers.

If that isn't the culprit, my next stop would definitely be looking at the 'Application' section of Event Viewer. It could be throwing some meaningful warnings to help you get closer to the root of the issue.

Julian Wolfe
  • 1
  • 2