.NET Remoting giving a KRB_AP_ERR_MODIFIED Error

Question

We have an error in our application, I hope someone can help. Here's a summary of what is happening:

1) Our application has a 3-tier architecture – a Silverlight application, a web tier and a backend business logic tier.

• The business logic tier (largely legacy code) runs as a Windows service.
• We have one service for each client.
• There are multiple business logic servers in the system, each running multiple Windows services for our clients
• The Silverlight browser application will communicate with the web tier via a set of WCF web services and the web tier will communicate with the business logic via .NET Remoting.
• Each client’s windows services will establish a TCP port for remoting endpoints when it starts.
• The role of the web tier is to get an incoming request and route it to the correct service.

2) We have a sporadic issue in our SilverLight application. Users will see a message in the interface that says either “Cannot open log for source 'ExceptionManagerInternalException'. You may not have write access” or “A call to SSPI failed, see inner exception.” Both errors are caused by the same issue, users with admin rights on the system will see the second message, 'normal' client users will see the first.

3) More details of the error can be seen in either the event logs on the web server. In the system event log, we see:

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the aaa$. The target name used was HOST/bbb.cab.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (domain.LOCAL) is different from the client domain (domain.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

4) We have captured trace files that have allowed us to see the events that occur when we get this bug:

• A SOAP call comes into the Web API (WCF)
• A client id is recovered from the http header. The client id allows us to uniquely identify a user working at an instance of a browser and was put into the header when the client first logged on to the system. When they logged on, we also cached the client id, together with the service that they connect to, on the web server. This is implemented in a WCF behaviour on our endpoint.
• Using this lookup, the address of the system (ie the name of the server and the tcp port) and the SPN of the service (HOST\) are put onto thread storage.
• We have used logging (NLOG) to confirm that this process is happening ok
• The code on the web tier will then make a .NET Remoting call to the server.
• The code in the web tier then makes the .NET Remoting call(s) to the business logic tier. In the ClientChannelSink, we get the address of the server and SPN from thread storage and configure the remoting call to go to the correct destination. NLOG traces have confirmed that the correct details are retrieved from thread storage and used to configure the call. We saw a trace where a request was made by UserA@domain.local to go to aaa using an SPN of HOST\aaa).
• From a network monitor trace, we can see DNS packages and the correct ip addresses are returned for the server names.
• From a network monitor trace, we can see that a Kerberos ticketing request is made to the DC.
• This request will contain the SPN for a different server than expected (in our trace, the request asked for HOST\bbb)
• The Kerberos ticketing service returns a valid Kerberos ticket for the user (UserA@domain.local) for the requested service (HOST\bbb)
• A request is sent to the expected service (server aaa), using the valid Kerberos ticket for the wrong service.
• The service (on server aaa) rejects the request. This is legitimate since the Kerberos ticket is for not for HOST\aaa but HOST\bbb

I don’t think any of the ‘advice’ in the event log is the cause of this issue: - We don’t have any duplicate SPNs on our system - Both computers are on the same domain - The issue is sporadic; the issues listed in the message look like they would cause total failure all of the time.

I have read about asp.net thread agility, but I don’t think that this is causing this issue, since we always send our request to the correct server, but with the wrong Kerberos ticket. If we were using the ‘wrong’ settings from thread storage, then we would send the request to the wrong server with a matching Kerberos ticket for that server. Any help to the cause or how to investigate this further would by much appreciated.

Answer 1

I'm not sure of your backend interface, but you shouldn't need to explicitly pass around the Kerberos ticket itself. You just need to create the windows principal and determine if you want to allow impersonation in your server. If so, then you must trust a given service for delegation by creating an SPN in the domain.

Do you have an SPN defined for the account and address of the business layer / data layer HOST\aaa server in the domain?

Your other option here is to bypass delegation altogether inside your business logic, perform a login on the front end and then use that to create your own security privileges within your app. That way you don't require the kerberos token to be passed around inside your business/data layers at all.