Demoted domain controller still authenticating users

Question

Why is a demoted domain controller still authenticating users?

Whenever users log onto workstations with domain accounts, this demoted DC authenticates them. Its security log shows their logons, logoffs, and special logons. Our new DCs' security logs show some machine logons and logoffs, but nothing to do with domain users.

Background

1. server1 (Windows Server 2008): Recently demoted DC, file server
2. server3 (Windows Server 2008 R2): New DC
3. server4 (Windows Server 2008 R2): New DC

Logs

Security Log Events: http://imgur.com/a/6cklL.

Two sample events from server1:

Audit Success,3/31/2014 11:06:14 AM,Microsoft-Windows-Security-Auditing,4624,Logon,"An account was successfully logged on.

Subject:
    Security ID:        NULL SID
    Account Name:       -
    Account Domain:     -
    Logon ID:       0x0

Logon Type:         3

New Logon:
    Security ID:        MYDOMAIN\auser
    Account Name:       auser
    Account Domain:     MYDOMAIN
    Logon ID:       0x8b792ce
    Logon GUID:     {54063226-E9B7-D357-AD58-546793C9CA59}

Process Information:
    Process ID:     0x0
    Process Name:       -

Network Information:
    Workstation Name:   
    Source Network Address: 192.168.20.143
    Source Port:        52834

Detailed Authentication Information:
    Logon Process:      Kerberos
    Authentication Package: Kerberos
    Transited Services: -
    Package Name (NTLM only):   -
    Key Length:     0

[ ... ]

Audit Success,3/31/2014 11:06:06 AM,Microsoft-Windows-Security-Auditing,4624,Logon,"An account was successfully logged on.

Subject:
    Security ID:        NULL SID
    Account Name:       -
    Account Domain:     -
    Logon ID:       0x0

Logon Type:         3

New Logon:
    Security ID:        MYDOMAIN\anotheruser
    Account Name:       anotheruser
    Account Domain:     MYDOMAIN
    Logon ID:       0x8b74ea5
    Logon GUID:     {7E74986A-7A4D-B6E0-5D6F-D8CF78E8C718}

Process Information:
    Process ID:     0x0
    Process Name:       -

Network Information:
    Workstation Name:   
    Source Network Address: 192.168.20.203
    Source Port:        53027

Detailed Authentication Information:
    Logon Process:      Kerberos
    Authentication Package: Kerberos
    Transited Services: -
    Package Name (NTLM only):   -
    Key Length:     0

Sample Audit Policy Change event from server3 (there are also Audit Policy Change events in the log with changes marked "Success Added"):

System audit policy was changed.

Subject:
    Security ID:        SYSTEM
    Account Name:       SERVER3$
    Account Domain:     MYDOMAIN
    Logon ID:       0x3e7

Audit Policy Change:
    Category:       Account Logon
    Subcategory:        Kerberos Authentication Service
    Subcategory GUID:   {0cce9242-69ae-11d9-bed3-505054503030}
    Changes:        Success removed

Attempted Solutions

1. Fixing DNS entries. dcdiag /test:dns at first returned errors after server1 was demoted. There were outdated name server entries in our forward lookup zones, for example. I ended up opening DNS Manager and removing the problem entries manually, also ensuring that LDAP and Kerberos entries pointed to the new servers. For instance, __ldap.Default-First-Site.__sites.dc.__msdcs.mydomain.local_ points to server3.mydomain.local.
2. Verifying DNS entries with nslookup. nslookup -type=srv _kerberos._udp.mydomain.local returns entries for server3 and server4—nothing about server1.
3. Cleaning up metadata. After using ntdsutil to clean up metadata as described in this TechNet article, the ntdsutil command list servers in site returns only two entries, which both look OK:
   1. 0 - CN=SERVER4,CN=Servers,CN=Default-First-Site,CN=Sites,CN=Configuration,DC=mydomain,DC=local
   2. 1 - CN=SERVER3,CN=Servers,CN=Default-First-Site,CN=Sites,CN=Configuration,DC=mydomain,DC=local
4. Deleting server1 from Active Directory Sites and Services. After demoting server1, I noticed it remained in Active Directory Sites and Services, although it was no longer listed as a global catalog. I deleted it according to the instructions in this Microsoft KB article.
5. Transferring operations master roles to server3. Operations master roles are a bit beyond my ken, but I used ntdsutil to transfer them all to server3 this morning. There were no errors, but reboots and tests showed that server1 was still doing all the authentication.
6. Reregistering with DNS and restarting netlogon. A forum post suggested running ipconfig /registerdns and net stop netlogon && net start netlogon on the new servers to resolve a related issue. It didn't seem to help.
7. Ensuring that the winning GPO on the new domain controllers enables auditing for logon and account logon events.

Other Leads

• The same problem is described in this series of forum posts. There's no resolution.
• It's also described in this question on Experts Exchange. The comment marked as an answer reads, "If the its [sic] not a DC anymore, then there is no way it can be processing any authentication requests." That would be my reaction, but running dcdiag on server1 confirms that server1 doesn't consider itself a DC. Yet it's still the only server authenticating everybody.

What's going on here?

Answer 1

It's a file server - are users connecting to it to get access to files? That's likely what you're seeing. Those would show up in the security logs.

Post some log entries (in their entirety - text dump or screenshot) from server1 that is showing the behavior you're concerned about.

/Edit - Thanks for confirming. Logon Type 3 is "Network." Most often seen when accessing shared files or printers on the computer that logged the event.

Answer 2

A demoted DC will in no way continue to authenticate domain logons. What you are seeing is local logon events. When you log onto a member server w/ domain credential, you will see logon events locally, plus corresponding credential validation events on DC.

When you log onto member server with local credential, you still see logon events locally, but won't see any credential validation events on DC.