3

It appears that there was a problem at some point that folks tracked down that caused authentication tickets to be marked invalid on a Webfarm when the servers had different mixes of patches.

UNFORTUNATELY, it seems that there is STILL a problem even when the web servers have identical patches on them.

My two servers:

  • have IDENTICAL patches
  • have identical machineKeys

Yet when a user transitions from one web server in the farm to the other web server the underlying microsoft code invalidates the token. Here's the Event log entry:

Event code: 4005 
Event message: Forms authentication failed for the request. Reason: The ticket supplied was invalid. 
Event time: 4/10/2012 2:42:20 PM 
Event time (UTC): 4/10/2012 6:42:20 PM 
Event ID: 92eedee52ede49239fd063fe5609d858 
Event sequence: 2 
Event occurrence: 1 
Event detail code: 50201 

Application information: 
    Application domain: /LM/W3SVC/2000/ROOT-1-129785553216092727 
    Trust level: Full 
    Application Virtual Path: / 
    Application Path: C:\inetpub\wwwroot\testportal.mydomain.com\ 
    Machine name: WEB02 

Process information: 
    Process ID: 1428 
    Process name: w3wp.exe 
    Account name: IIS APPPOOL\testportal.mydomain.com 

Request information: 
    Request URL: http://testportal.mydomain.com/Interface.aspx 
    Request path: /Interface.aspx 
    User host address: ************* 
    User:  
    Is authenticated: False 
    Authentication Type:  
    Thread account name: IIS APPPOOL\testportal.mydomain.com 

Name to authenticate:  

Custom event details:

I've either got to figure this out or write microsoft's authentication out of my system (which I really don't have time to do).

Here are the hotfixes applied IDENTICALLY to both servers which are:

Windows Server 2008 R2 x64-based

KB981391,KB981392,KB977236,KB981111,KB977238,KB977239,KB981390,KB2305420,KB2386667,KB2393802,KB2425227,KB2475792,KB2476490,KB2478662,KB2479628,KB2482017,KB2484033,KB2485376,KB2487426,KB2488113,KB2492386,KB2503665,KB2505438,KB2506014,KB2506212,KB2506928,KB2507618,KB2507938,KB2508272,KB2509553,KB2510531,KB2511250,KB2511455,KB2515325,KB2518869,KB2522422,KB2524375,KB2529073,KB2530548,KB2533552,KB2533623,KB2534366,KB2536275,KB2536276,KB2539635,KB2541014,KB2544521,KB2544893,KB2545698,KB2547666,KB2552343,KB2555917,KB2556532,KB2560656,KB2563227,KB2564958,KB2567680,KB2570947,KB2572077,KB2584146,KB2585542,KB2588516,KB2598845,KB2603229,KB2607047,KB2607576,KB2608658,KB2618444,KB2618451,KB2620704,KB2620712,KB2621440,KB2631813,KB2632503,KB2633873,KB2633952,KB2636573,KB2639308,KB2639417,KB2640148,KB2641653,KB2641690,KB2643719,KB2644615,KB2645640,KB2647516,KB2647518,KB2654428,KB2656356,KB2660075,KB2660465,KB2665364,KB2667402,KB958488,KB976902,KB976932,KB982018

Per request, here's the authentication section of my web.config:

    <authentication mode="Forms">
        <forms loginUrl="Login.aspx" name=".ASPXFORMSAUTH" cookieless="AutoDetect" timeout="120" slidingExpiration="true">
        </forms>
    </authentication>
TechSavvySam
  • 1,382
  • 16
  • 28
  • 1
    One thing you have failed to provide is your `` section in your web.config. That might highlight if you have any settings that should be changed. – Josh Apr 10 '12 at 19:05
  • Are you sure that the machineKey element (and not just some of it's attributes) are equal? Have you specified encryption/validation algorithms? – sisve Apr 11 '12 at 16:00
  • yep. I went back and double checked them when I started trying to diagnose this problem even though viewstate has been working fine across the webfarm. – TechSavvySam Apr 11 '12 at 16:07

4 Answers4

3

I think I have finally resolved this issue. I'm not exactly sure why this change fixes the problem but what I did to resolve it was to put the machineKey definition directly in the web.config file for my application. Apparently either I don't understand how to use IIS Manager to set up the machine keys properly or there is some issue with the IIS Manager.

So to solve the problem I put an entry directly in my web.config that looks something like this:

<machineKey validation="SHA1" validationKey="-a-validation-key-" decryption="Auto"  decryptionKey="-a-decryption-key-"/>

This article kind of led me in the correct path to solve this issue:

ASP.NET 4 Breaking Changes

TechSavvySam
  • 1,382
  • 16
  • 28
  • UPDATE: I've been running with this for 3 weeks now and it seems to have resolved the problem. – TechSavvySam Jul 03 '12 at 17:08
  • Did you include ",IsolateApps" in your validation / decryption keys or not? Did you do this in the web.config or the machine.config? Did you stick with SHA1 or use something else? I have the same problem on a webfarm running .net 4.0. Havent fixed it yet, have tried putting the same machinekey in all servers web.configs. But no luck so far – JK. Aug 25 '12 at 00:06
  • Also, is there anything you need to do after changing the machinekey in we.config to make it "take" eg run iisreset etc? – JK. Aug 25 '12 at 00:18
  • 1
    The XML snippet above is exactly what I used (except I removed the keys from the XML above), so no I did not use IsolateApps. I did it in the web.config files--I try not to make changes in the machine.config plus my web.config files are source code controlled. If it still doesn't work for you, you might have one of the other issues such as mismatched patch levels between servers. – TechSavvySam Aug 28 '12 at 00:12
2

Well I spoke too soon. During this whole process, I deleted .NET Framework 4 off of the server because we are not developing to it and I had seen some mention of FW4 causing authentication issues.

After "resolving" this issue I put Framework 4 back on the servers and ran windows update which added the following patches to both servers:

  • Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2600217)
  • Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2533523)
  • Security Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2487367)
  • Security Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2518870)
  • Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2656351)
  • Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2572078)
  • Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2633870)
  • Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2468871)
  • Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2656368)

Since I made these updates, the problem is now back. The other interesting thing is that somewhere during the process, these updates chopped the decryption key back to 48 bytes. I'm not sure if it chopped the existing one or if it created a new one. I went back and put a newly generated Machine Key and Decryption Key and it did not fix the problem.

TechSavvySam
  • 1,382
  • 16
  • 28
  • 1
    Looks like I'm not the only one that is about to give up on Microsoft .NET Authentication as being permanently broken: http://blog.appharbor.com/2012/02/22/asp-net-forms-authentication-considered-broken – TechSavvySam May 03 '12 at 20:36
1

Well with some help from this guy over at asp.net I figured out how to fix the issue:

Thread at asp.net that helped me figure this out.

I think what finally fixed it was regenerating the Validation and Decryption keys. There must have been something about the old keys that was incompatible with the hotfixes loaded on these servers.

On an interesting note, the old and new validation keys were both 128 bytes but the old Decryption Key was 48 bytes and the new one is 64 bytes.

TechSavvySam
  • 1,382
  • 16
  • 28
0

I've seen this behavior when MS10-070 (asp.net padding oracle attack) was applied to some of the servers. This update changes [in some way] the encryption technique, making the tickets incompatible with unpatched servers within the same farm.

Can you verify that all servers in your farm has this update applied?

sisve
  • 19,501
  • 3
  • 53
  • 95
  • For Windows Server 2008 R2 that would be the following KB numbers: Microsoft .NET Framework 3.5.1* (KB2416471) Microsoft .NET Framework 4.0[1] (KB2416472) and neither one of the servers have that on it (see updated comment above) – TechSavvySam Apr 11 '12 at 02:05
  • On another issue; why dont they have an important security update applied? – sisve Apr 11 '12 at 05:39
  • Probably because not all security patches apply to all configurations (for example a more recently built system will have a patch that supersedes the original patch so the patch you're talking about may have never been required). Trust me, the VERY first thing I did when I started having this problem was run windows update an make both servers current. – TechSavvySam Apr 11 '12 at 13:48
  • 7 new patches available as of yesterday I guess--I'll update both servers and see if this improves anything. – TechSavvySam Apr 11 '12 at 13:51
  • 1
    And thank goodness this webfarm isn't in production yet--I would have had to revert them to a functioning state and then I'm not clear how I would ever be able to update them w/o a duplicate webfarm to test on. Microsoft's shoddy quality continues to amaze me--if we wrote software this bad and were this unresponsive to client's complaints we would be out of business... – TechSavvySam Apr 11 '12 at 13:57
  • Simon, here's the answer to your question: Under MS10-070 it lists "non-affected systems" to include: Windows Server 2008 R2 for x64-based Systems Service Pack 1 -Microsoft .NET Framework 3.5.1 (which is why my system doesn't have it) – TechSavvySam Apr 11 '12 at 14:07
  • Latest patches did not fix the issue – TechSavvySam Apr 11 '12 at 14:23