Remote endpoint has failed - happens randomly

Question

We have a setup with a website, which communicates using WCF to our Microsoft CRM 2011 solution.

Users create new cases on our website, which are stored in the CRM solution. Using WCF, it calls CRM and stores the data.

This works perfectly, but some few times a day the following error comes. We cannot find anything unique about these occurrences: Random users, random input, ect. There are nothing which seems unique for the users who get the error.

We can see in our IIS event viewer the error happens ~3 times a day, depending on how many users we have online.

Any ideas? What would be unique? Hints? Something? :-) Thanks!

The error:

Exception information: 
    Exception type: SecurityNegotiationException 
    Exception message: Secure channel cannot be opened because security negotiation with the remote endpoint has failed. This may be due to absent or incorrectly specified EndpointIdentity in the EndpointAddress used to create the channel. Please verify the EndpointIdentity specified or implied by the EndpointAddress correctly identifies the remote endpoint. 

Server stack trace: 
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Our setup:

We have 1 production website server and CRM server. There are no load balancer or anything else.

WCF binding

 <system.serviceModel>
        <bindings>
            <customBinding>
                <binding name="CustomBinding_IOrganizationService">
                    <security defaultAlgorithmSuite="Default" authenticationMode="SspiNegotiated"
                        requireDerivedKeys="true" securityHeaderLayout="Strict" includeTimestamp="true"
                        keyEntropyMode="CombinedEntropy" messageProtectionOrder="SignBeforeEncryptAndEncryptSignature"
                        messageSecurityVersion="WSSecurity11WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10"
                        requireSecurityContextCancellation="true" requireSignatureConfirmation="false">
                        <localClientSettings cacheCookies="true" detectReplays="true"
                            replayCacheSize="900000" maxClockSkew="00:05:00" maxCookieCachingTime="Infinite"
                            replayWindow="00:05:00" sessionKeyRenewalInterval="10:00:00"
                            sessionKeyRolloverInterval="00:05:00" reconnectTransportOnFailure="true"
                            timestampValidityDuration="00:05:00" cookieRenewalThresholdPercentage="60" />
                        <localServiceSettings detectReplays="true" issuedCookieLifetime="10:00:00"
                            maxStatefulNegotiations="128" replayCacheSize="900000" maxClockSkew="00:05:00"
                            negotiationTimeout="00:01:00" replayWindow="00:05:00" inactivityTimeout="00:02:00"
                            sessionKeyRenewalInterval="15:00:00" sessionKeyRolloverInterval="00:05:00"
                            reconnectTransportOnFailure="true" maxPendingSessions="128"
                            maxCachedCookies="1000" timestampValidityDuration="00:05:00" />
                        <secureConversationBootstrap />
                    </security>
                    <textMessageEncoding maxReadPoolSize="64" maxWritePoolSize="16"
                        messageVersion="Default" writeEncoding="utf-8">
                        <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                            maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                    </textMessageEncoding>
                    <httpTransport manualAddressing="false" maxBufferPoolSize="524288"
                        maxReceivedMessageSize="65536" allowCookies="false" authenticationScheme="Anonymous"
                        bypassProxyOnLocal="false" decompressionEnabled="true" hostNameComparisonMode="StrongWildcard"
                        keepAliveEnabled="true" maxBufferSize="65536" proxyAuthenticationScheme="Anonymous"
                        realm="" transferMode="Buffered" unsafeConnectionNtlmAuthentication="false"
                        useDefaultWebProxy="true" />
                </binding>
                <binding name="CustomBinding_IOrganizationService1">
                    <security defaultAlgorithmSuite="Default" authenticationMode="SspiNegotiated"
                        requireDerivedKeys="true" securityHeaderLayout="Strict" includeTimestamp="true"
                        keyEntropyMode="CombinedEntropy" messageProtectionOrder="SignBeforeEncryptAndEncryptSignature"
                        messageSecurityVersion="WSSecurity11WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10"
                        requireSecurityContextCancellation="true" requireSignatureConfirmation="false">
                        <localClientSettings cacheCookies="true" detectReplays="true"
                            replayCacheSize="900000" maxClockSkew="00:05:00" maxCookieCachingTime="Infinite"
                            replayWindow="00:05:00" sessionKeyRenewalInterval="10:00:00"
                            sessionKeyRolloverInterval="00:05:00" reconnectTransportOnFailure="true"
                            timestampValidityDuration="00:05:00" cookieRenewalThresholdPercentage="60" />
                        <localServiceSettings detectReplays="true" issuedCookieLifetime="10:00:00"
                            maxStatefulNegotiations="128" replayCacheSize="900000" maxClockSkew="00:05:00"
                            negotiationTimeout="00:01:00" replayWindow="00:05:00" inactivityTimeout="00:02:00"
                            sessionKeyRenewalInterval="15:00:00" sessionKeyRolloverInterval="00:05:00"
                            reconnectTransportOnFailure="true" maxPendingSessions="128"
                            maxCachedCookies="1000" timestampValidityDuration="00:05:00" />
                        <secureConversationBootstrap />
                    </security>
                    <textMessageEncoding maxReadPoolSize="64" maxWritePoolSize="16"
                        messageVersion="Default" writeEncoding="utf-8">
                        <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                            maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                    </textMessageEncoding>
                    <httpTransport manualAddressing="false" maxBufferPoolSize="524288"
                        maxReceivedMessageSize="65536" allowCookies="false" authenticationScheme="Anonymous"
                        bypassProxyOnLocal="false" decompressionEnabled="true" hostNameComparisonMode="StrongWildcard"
                        keepAliveEnabled="true" maxBufferSize="65536" proxyAuthenticationScheme="Anonymous"
                        realm="" transferMode="Buffered" unsafeConnectionNtlmAuthentication="false"
                        useDefaultWebProxy="true" />
                </binding>
            </customBinding>
        </bindings>
        <client>
            <endpoint address="http://OURDOMAIN/XRMServices/2011/Organization.svc"
                binding="customBinding" bindingConfiguration="CustomBinding_IOrganizationService"
                contract="CRMService.IOrganizationService" name="CustomBinding_IOrganizationService">
                <identity>
                    <servicePrincipalName value="host/OURDOMAIN" />
                </identity>
            </endpoint>
            <endpoint address="http://OURDOMAIN/XRMServices/2011/Organization.svc"
                binding="customBinding" bindingConfiguration="CustomBinding_IOrganizationService1"
                contract="XRMService.IOrganizationService" name="CustomBinding_IOrganizationService1">
                <identity>
                    <servicePrincipalName value="host/OURDOMAIN" />
                </identity>
            </endpoint>
        </client>
    </system.serviceModel>

I recommend posting some code showing how you instantiate CRM service - your issue may not lie in config. — Greg Owens, Sep 17 '12 at 13:23

score 1 · Accepted Answer · answered Oct 23 '12 at 13:42

The answer was quite interesting. We found out a team at our company ran some extremely heavy testing towards our CRM system from time to time. That blocked our call and resulted in the timeout.

That could also explain the randomness of the action.

We found out by enabling tracing on the computer, so we could see a graph of the performance.

Remote endpoint has failed - happens randomly

1 Answers1