4

We have a very basic SOAP web service setup using Windows Authentication, open for all users:

<authentication mode="Windows" />
<authorization>
  <allow users="*" />
</authorization>

The Issue

However, some Windows accounts are getting 401 Forbidden errors.

What works

  • A select list of accounts always work (regardless of NTLM/Kerberos, local/external server)
  • All accounts seem to work when accessing locally on the web server
  • All accounts seem to work when using IE (which seems to use Kerberos)

What doesnt work

  • Most (but not all) accounts from an external server (which seems to use NTLM)

Other Facts

  • There is no obvious pattern to which accounts work and which don't (it seems arbitrary).
  • A new account identical to an existing working account does not work.
  • Which accounts work seems to be consistent (i.e. it will either always work or never work)
  • The accounts which work seem consistent across environments (i.e. all IIS servers accept and reject the same set of accounts).
  • There's no differences in the captured traffic between the servers, beyond different NTLM tokens and a 401 instead of 200/202 response.

What Happens Technically

Specifically, when an issue occurs it as though the password credentials cannot be verified (I can guarantee they are correct), as per the 4625 Audit Failure error in the Security log:

Account For Which Logon Failed:
Security ID:        NULL SID
Account Name:       FAILING_ACCOUNT_NAME_ALLCAPS
Account Domain:     MYDOMAIN

Failure Information:
Failure Reason:     Unknown user name or bad password.
Status:         0xc000006d
Sub Status:     0xc000006a

This seems to lead to the account being locked out in AD (which would makes sense as the 0x000006a code usually reflects an invalid password attempt), although this behaviour seems inconsistent.

With failed request tracing, IIS shows the unknown user name/bad password error, as well as a 401.2 error with the error code 3221225581.

I can also force "genuine" 401 errors for working accounts (by denying them in the authorization tag) and the behaviour differs (i.e. it does not flag these as Unknown user name or bad password but gives a different access denied error).

What Next

You tell me - does anyone have experience with where I should look next to help diagnose this issue?

Matt Mitchell
  • 40,943
  • 35
  • 118
  • 185
  • You need to provide your domain environment. I assume there are at least two AD domains. You need to provide the domain trust relationship. Is it one-way trust or two-way trust? Is it external trust or forest trust? I suggest you look at the network trace to see what's really going on. There are too many *seems to be*. Attach a network capture may also help others to find out the cause of the problem. – Harvey Kwok Mar 24 '12 at 19:43
  • Single AD domain. Packet capture is identical exception for different tokens. Microsoft support are currently working with us to resolve through a lot of packet capture and no explanation so far. – Matt Mitchell Apr 26 '12 at 05:09
  • 2
    Did you ever figure this out? I'm seeing the same entry in the Security log with the same Status and Sub Status. – Sean Hall Aug 20 '12 at 18:44
  • Nope. We did find that for one particular account it was related to the password. We could set a simpler password and it would work, then set back to a more complex and it wouldn't. Setting this same more complex password for another user caused an issue. However, this wasn't reliable for all users (many of which had simple passwords). We've had Microsoft review our logs and they can't determine a reliable cause yet either (few months of investigation). Interestingly, and probably unrelated, during Windows updates we find a similar issue even authenticating against the local server. – Matt Mitchell Aug 20 '12 at 23:30

1 Answers1

1

Seems to be an extremely rare issue that haunts you for years (infrastructure teams, developers, and Microsoft have been unable to isolate the issue). As per comments it seems particular passwords can have an effect so try there.

Matt Mitchell
  • 40,943
  • 35
  • 118
  • 185