I do not understand WCF scaling with security / Why is lsass.exe CPU% so high?

Question

I've got a WCF service with about 500 clients that make a call every 3 minutes. The lsass.exe process is using 95% of my CPU when the service is on.
I did a test and every time a client makes a call, the lsass CPU% goes up to around 7 or so. So I understand why the server is slowing to a call when a couple hundred are sending calls at the same time.
What I don't understand, is why is lsass being used at all? I'm using a CustomBinding based off of the WSHttpBinding with Message level security using Certificates. Instance/Concurrency is PerCall/Single (though I've tried pretty much every combination of the two with no change), and it's hosted in IIS 6.
My guess is that for every call, lsass is checking the certificate, and this somehow takes a lot of CPU? Is there some way I can lessen that? I know people have scaled WCF services to much larger numbers, so what is it that I'm doing wrong?

P.S. Some extra info: I've got tracing on and the first thing that usually gets logged is

<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Warning">
    <TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Activation.WebHostNoCBTSupport.aspx</TraceIdentifier>
        <Description>Extended protection is not supported or not enabled on this platform. Please install the appropriate patch and enable it if you want extendedProtection support for https with windows authentication.</Description>
        <AppDomain>/LM/W3SVC/920256058/Root/WCFServices/smt-7-129642078492968750</AppDomain>
        <Source>System.ServiceModel.Activation.MetabaseSettingsIis6/17731154</Source>
</TraceRecord>

And there's also a bunch of:

http://msdn.microsoft.com/en-US/library/System.ServiceModel.Security.SecuritySessionDemuxFailure.aspx
The incoming message is not part of an existing security session.

and

<ExceptionType>System.ServiceModel.Security.SecurityNegotiationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>Cannot find the negotiation state for the context 'uuid-4caa56ff-3d38-4905-9151-ce12acdd676c-6778'.</Message>

and

<NegotiationTokenAuthenticator>System.ServiceModel.Security.TlsnegoTokenAuthenticator</NegotiationTokenAuthenticator>
<AuthenticatorListenUri>http://myserviceendpoint</AuthenticatorListenUri>
<Exception>System.ServiceModel.Security.SecurityNegotiationException: Cannot find the negotiation state for the context 'uuid-4caa56ff-3d38-4905-9151-ce12acdd676c-6778'.at System.ServiceModel.Security.NegotiationTokenAuthenticator`1.ProcessRequestCore(Message request) at System.ServiceModel.Security.NegotiationTokenAuthenticator`1.NegotiationHost.NegotiationSyncInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp;amp; outputs)</Exception>

My IIS log is full of

Mostly the last two

Edit: Service config:

<?xml version="1.0"?>
<configuration>
  <configSections>
    <section name="dataConfiguration"     type="Microsoft.Practices.EnterpriseLibrary.Data.Configuration.DatabaseSettings,     Microsoft.Practices.EnterpriseLibrary.Data"/>
  </configSections>
  <connectionStrings>
    <add name="conString" connectionString="Data Source=source1;Failover     Partner=source2;database=myDatabase;Integrated Security=false;uid=userName;pwd=password"     providerName="System.Data.SqlClient"/>
  </connectionStrings>
  <dataConfiguration defaultDatabase="conString"/>
  <system.web>
    <compilation debug="true" targetFramework="4.0"/>
    <authorization>
      <allow users="?"/>
    </authorization>
  </system.web>

  <system.serviceModel>
    <services>
      <service name="ServiceLibrary.Service">
        <endpoint address="ws" binding="wsHttpBinding"     bindingConfiguration="WSHttpBinding_IService"
          name="WSHttpEndpoint_IService"     contract="ServiceLibrary.IService" />
        <endpoint address="cs1" binding="customBinding"     bindingConfiguration="CustomBinding_IService"
          name="CustomEndpoint_IService"     contract="ServiceLibrary.IService" />
        <endpoint address="basic" binding="basicHttpBinding"     name="BasicHttpEndpoint_IService"
          contract="ServiceLibrary.IService" />
        <endpoint address="mex" binding="mexHttpBinding"
          contract="IMetadataExchange" />
      </service>
    </services>
    <bindings>
      <wsHttpBinding>
        <binding name="WSHttpBinding_IService"
            maxBufferPoolSize="524288" maxReceivedMessageSize="1048576">
          <readerQuotas maxDepth="32" maxStringContentLength="65536" maxArrayLength="16384" 
              maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          <security mode="Message">
            <message clientCredentialType="Certificate" negotiateServiceCredential="true"
                algorithmSuite="Default" />
          </security>
        </binding>
      </wsHttpBinding>
      <customBinding>
        <binding name="CustomBinding_IService">
          <transactionFlow />
          <security authenticationMode="SecureConversation"     messageSecurityVersion="WSSecurity11WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10">
            <localServiceSettings maxClockSkew="00:10:00" maxPendingSessions="102400" />
            <localClientSettings maxClockSkew="00:10:00" />
            <secureConversationBootstrap authenticationMode="MutualSslNegotiated"     messageSecurityVersion="WSSecurity11WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10" />
          </security>
          <textMessageEncoding>
            <readerQuotas maxDepth="32" maxStringContentLength="65536" maxArrayLength="16384"     maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          </textMessageEncoding>
          <httpTransport maxBufferSize="1048576" maxReceivedMessageSize="1048576" />
        </binding>
      </customBinding>
    </bindings>
    <behaviors>
      <serviceBehaviors>
        <behavior>
          <serviceCredentials>
            <serviceCertificate findValue="CN=[domain]" storeLocation="LocalMachine"     storeName="TrustedPeople" />
            <clientCertificate>
              <authentication revocationMode="NoCheck" certificateValidationMode="PeerTrust"     />
            </clientCertificate>
          </serviceCredentials>
          <serviceThrottling maxConcurrentCalls="1000"  maxConcurrentSessions="1000"     maxConcurrentInstances="1000" />
          <serviceMetadata httpGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="false"/>
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <serviceHostingEnvironment multipleSiteBindingsEnabled="true"/>
  </system.serviceModel>
</configuration>

The clients are all using the CustomBinding. The problem still occurs with the WS binding though. In fact I made the CustomBinding to solve a problem where the service was running out of pending sessions.

Show configuration of your service as well. – Ladislav Mrnka Oct 28 '11 at 11:53 — Ladislav Mrnka, Oct 28 '11 at 11:53

score 0 · Answer 1 · answered Oct 28 '11 at 22:04

So, I finally got everything working again, more or less. My problem was that all of the clients were trying to make calls at once, and it just overloaded the server. I set the IIS application to deny all incoming traffic, then added groups of IPs slowly back in until finally they were all able to connect.

Lsass is still using a high amount of CPU, which is unacceptable. I'm going to have to find a different security method (other than certificates) which is faster and doesn't make lsass go crazy because we have hundreds of clients.

I do not understand WCF scaling with security / Why is lsass.exe CPU% so high?

1 Answers1