4

I'm developing a website for a client of ours and have been having a nagging issue for weeks. The problem is intermittent and might happen twice in a day or once in 2 weeks.

When it happens, the client will start getting 401.3 "Unauthorized" errors when trying to load static script or style tags on the page. It's always a .css or .js file that gets denied and setting <modules runAllManagedModulesForAllRequests="false" /> hasn't fixed the problem. When the problem happens it effectively blocks the file until the application pool is restarted.

The site is using Forms Authentication with the stock "AspNetSqlMembershipProvider". The site has protected files in an /Accounts folder, styles in /Styles, and scripts in /Scripts. The site has anonymous access enabled and has a separate web.config in the /Accounts folder which deny's access to anonymous users (though that shouldn't even matter with runAllManagedModulesForAllRequests=false).

I've looked through the event logs and I see this:
2012-06-06 14:23:32 [ipaddress] GET /subfolder/Styles/Site.css ver=11 443 - [ipaddress] Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:12.0)+Gecko/20100101+Firefox/12.0 https://thesitename.net/subfolder/Account/Login.aspx 401 3 5 1493 429 62

Sometimes after a short period of time it fixes itself or if I restart the application pool I get this:
2012-06-06 14:30:30 [ipaddress] GET /subfolder/Styles/Site.css ver=11 443 - [ipaddress] Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:12.0)+Gecko/20100101+Firefox/12.0 https://thesitename.net/subfolder/Account/Login.aspx 200 0 0 2533 455 78

I finally got the hosting provider to enable failed request tracing and finally got a trace of the error: frt

Every failed request tracing file I got this time said the failure happens right after the "FILE_CACHE_ACCESS_END" event. What is this event doing and why would access be denied?

Jim Brown
  • 498
  • 1
  • 6
  • 20
  • Can you add an example line from the log indicating one of these errors? – Chris Shain Jun 11 '12 at 16:45
  • Chris, I added two lines from the IIS log. One is when the problem is happening and then a second line showing the response after the application pool has been restarted. – Jim Brown Jun 11 '12 at 21:00

3 Answers3

1

EDIT:

You should probably try following the "failed request tracing" instructions here. The error is almost certainly NTFS permissions related.

Looks like you are getting a Win32 "path not found" error in the problem line (the 3 right after the HTTP status code, field sc-win32-status).

Are these files on a network share? It looks like there might be some connectivity issues between the web server and the drive hosting these files. This might be in the form of an authorization error for the IIS user when attempting to read those files, which would explain the 401 error.

W3C Extended Log File Format (IIS 6.0) here: http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/676400bc-8969-4aa7-851a-9319490a9bbb.mspx?mfr=true

Win32 status codes here: http://msdn.microsoft.com/en-us/library/ms681382.aspx

Chris Shain
  • 50,833
  • 6
  • 93
  • 125
  • Note that I may be completely off here if you have fields in a different order. Can you confirm the field order? – Chris Shain Jun 11 '12 at 21:40
  • I copied it directly out of the log, but I never noticed that it was a 401.3 and what that might imply. Since the site is hosted and I have no control or insight to their environment I've created a support ticket for them to investigate this new information. I still don't understand why connectivity issues would be resolved by restarting the application pool though? I'll let you know once I have more information. Thank you for your help so far! – Jim Brown Jun 12 '12 at 15:50
  • After some research it looks like the sc-status is 401, the sc-substatus is 3, and the sc-win32-status is 5. The hosting services says the files are not on a network share and are hosted locally to the server so that's not it. – Jim Brown Jun 13 '12 at 16:43
  • @JimBrown added a link above that should give you some steps to try. – Chris Shain Jun 13 '12 at 18:17
  • I finally got the hosted provider to enable failed request tracing for me (they don't allow it normally). I also added health monitoring and am logging events to a database so between the two of those hopefully I can gather more information and post it back here. – Jim Brown Jun 14 '12 at 17:41
  • Even though it's not on a network drive, you've helped more than anyone else and have hopefully brought attention to this question. Hopefully with your help I can fix this... – Jim Brown Jun 18 '12 at 17:13
  • Has failed request tracing not brought any clarity? – Chris Shain Jun 18 '12 at 17:29
  • No, the hosting provider limited the number of files output to 50 so by the time I went to get them they had already been overwritten... – Jim Brown Jun 18 '12 at 17:46
  • Ouch. Maybe worth writing a little script to run on the server and keep fetching them until you see a 401, then grab the current file and email it to yourself. – Chris Shain Jun 18 '12 at 17:49
  • Okay, I finally got a good trace of global.js getting a 401.3. However, I still don't understand WHY it would suddenly get it and then suddenly let it pass again? – Jim Brown Jun 22 '12 at 18:11
  • Does your site experience extremely high traffic around the time that these issues begin to occur? Also, are your users defined locally on the machine, or are they AD users? – Chris Shain Jun 22 '12 at 18:16
  • No, actually it was pretty low levels of traffic. Only about 10 requests/minute in the IIS log. The users are defined in a standard sql server auth DB. – Jim Brown Jun 22 '12 at 19:14
  • I mean the *Windows* users, not the website users. Specifically, the Windows account that IIS runs as. – Chris Shain Jun 22 '12 at 19:18
  • I had to look through my health monitoring logs since the hosting provider won't give me this information :) It's WH001\IISSvc0_[sitename]. So they use some custom domain account it appears. – Jim Brown Jun 22 '12 at 19:39
  • I added the cacls and icacls for the file in question. – Jim Brown Jun 22 '12 at 19:49
  • Correction, it appears in the trace that the application user is IIS_ISR_[sitename] – Jim Brown Jun 22 '12 at 20:13
  • Chris, I have new information in my question, the error happens after the FILE_ACCESS_END event. Thanks. – Jim Brown Jul 06 '12 at 15:21
  • Its clearly an access denied error. I'd have the web host company confirm that this is a standalone windows server, because the only thing that comes to mind is communication issues between this server and a domain controller. Could even be a clock drift issue- see http://cryptnet.net/fdp/admin/kerby-infra/en/kerby-infra.html#time-sync-importance – Chris Shain Jul 06 '12 at 15:28
  • The hosting provider says this is a standalone server and they don't believe clock drift is an issue but they will look into it anyways... – Jim Brown Jul 06 '12 at 23:55
  • Is it possible at all that something in the application is causing these files to be locked? Recycling the application pool fixes it but that could just be something wrong with IIS right? I'm trying to rule out our code and our changes to the application as the cause for this error... – Jim Brown Jul 09 '12 at 17:26
  • I never did find out what is causing this but I don't have access to the server. I ended up recommending that we change hosting providers (which may fix the problem anyways) to one with a virtual server that I can remote into and run diagnostics if needed. Thanks for all of your help! – Jim Brown Aug 09 '12 at 18:55
0

I would look into the reason for that error in the health monitoring logs (event code 3005: System.Data.Linq.ChangeConflictException) before anything else. If your problems start to occur after this error, then there is a good chance that eliminating this error will resolve your problems.

Here are some pages that discuss this error and might be of help to understand the error and how to fix it:

http://blogs.microsoft.co.il/blogs/bursteg/archive/2007/11/25/linq-to-sql-concurrency-changeconflictexception-row-not-found-or-changed.aspx

http://sebastienlachance.com/post/SystemDataLinqChangeConflictException-Row-not-found-or-changed.aspx

http://weblogs.asp.net/okloeten/archive/2008/04/28/6139181.aspx

paulH
  • 1,102
  • 16
  • 43
  • Actually, this exception turned out to be nothing. The reason I was getting the denial error is because I told it to in the configuration for testing and forgot about it. I'm going to re-edit the question to show a lot of new evidence I've found since. – Jim Brown Jun 18 '12 at 16:47
0

I've been struggling with this error for longtime.

One of the strange causes of this error is when the resources (files or folders) are in a share (SMB share) with limited or no access rights to "everyone". This overrides the rights you may specifically add to the folder or files and causes a 401.3 error for anonymous access.

Marc
  • 16
  • 2
  • Thanks for sharing. Unfortunately this was a shared hosted server and I had no visibility to where files were stored. – Jim Brown Jun 07 '13 at 17:45