WCF NetNamedPipeBinding Delay On Channel Open

Question

I'm currently developing an application with a SOA architecture, with the services exposed as WCF services (.Net 4.0) hosted in IIS 7.5 on a Windows Server 2008 R2 Datacenter x64 VM (it is in fact an m1.small instance on Amazon EC2). These services talk to each other locally on the machine and as such I have set them to use netNamedPipeBinding for optimal performance. Instancing mode is per call and concurrency is set to multiple.

I'm hitting two issues at the moment whereby I'm getting intermittent delays when opening the channels of between 200ms to 1s, which is unacceptable as normal speed seems to be ~2ms.

I've enabled WCF tracing and I'm seeing that the delays manifest themselves as either an error of:

System.IO.PipeException: There was an error writing to the pipe: The pipe is being closed. (232, 0xe8).

after which it appears WCF retries and successfully connects (hence the delay). The second symptom is a half second delay when performing the activity:

Process action 'http://tempuri.org/IConnectionRegister/ValidateUriRoute'

The only thing I can find around this is that some people think it may be related to TCP port sharing, but I'm using named pipes. I tried disabling the TCP Port Sharing service but this made no difference.

Out of interest I have also tried changing all the endpoints to use net.tcp listening on localhost on the same random port, and the half second delay within the ValidateUriRoute activity still intermittently occurred.

My WCF configuration looks similar to this:

<system.serviceModel>

    <serviceHostingEnvironment aspNetCompatibilityEnabled="true"
                               multipleSiteBindingsEnabled="false">

      <serviceActivations>

        <add relativeAddress="ConfigurationHost.svc" service="Core.ConfigurationHost" factory="Core.ConfigurationHostFactory" />
        <add relativeAddress="RoutingHost.svc" service="Core.RoutingHost" factory="Core.RoutingHostFactory" />
        <add relativeAddress="AuthenticationHost.svc" service="Core.AuthenticationHost" factory="Core.AuthenticationHostFactory" />

      </serviceActivations>

    </serviceHostingEnvironment>

    <services>

      <service name="Core.ConfigurationHost"
               behaviorConfiguration="Unthrottled">

        <endpoint address="net.pipe://localhost/ConfigurationHost.svc"
                  binding="netNamedPipeBinding"
                  bindingConfiguration="customNetNamedPipeBinding"
                  contract="Core.IConfiguration" />

      </service>

      <service name="Core.RoutingHost"
               behaviorConfiguration="Unthrottled" >

        <endpoint address="net.pipe://localhost/RoutingHost.svc"
                  binding="netNamedPipeBinding"
                  bindingConfiguration="customNetNamedPipeBinding"
                  contract="Core.IRouting" />

      </service>

      <service name="Core.AuthenticationHost"
               behaviorConfiguration="Unthrottled">

        <endpoint address="net.pipe://localhost/AuthenticationHost.svc"
                  binding="netNamedPipeBinding"
                  bindingConfiguration="CustomNetNamedPipeBinding"
                  contract="Core.IAuthentication" />

      </service>

    </services>

    <behaviors>

      <serviceBehaviors>

        <behavior  name="Unthrottled">

          <serviceThrottling maxConcurrentCalls="100"
                             maxConcurrentSessions="100"
                             maxConcurrentInstances="100" />

        </behavior>

      </serviceBehaviors>

    </behaviors>

    <client>

      <endpoint address="net.pipe://localhost/ConfigurationHost.svc"
                binding="netNamedPipeBinding"
                bindingConfiguration="customNetNamedPipeBinding"
                contract="Core.IConfiguration"
                name="Configuration" />

      <endpoint address="net.pipe://localhost/RoutingHost.svc"
                binding="netNamedPipeBinding"
                bindingConfiguration="customNetNamedPipeBinding"
                contract="Core.IRouting"
                name="Routing" />

      <endpoint address="net.pipe://localhost/AuthenticationHost.svc"
                binding="netNamedPipeBinding"
                bindingConfiguration="customNetNamedPipeBinding"
                contract="Core.IAuthentication"
                name="Authentication" />

    </client>

    <bindings>

      <netNamedPipeBinding>

        <binding name="customNetNamedPipeBinding"
                 maxReceivedMessageSize="2147483647"
                 sendTimeout="00:00:30"
                 receiveTimeout="infinite"
                 closeTimeout="00:00:30"
                 openTimeout="00:00:30"
                 maxConnections="500">

          <security mode="None"/>

          <readerQuotas maxDepth="200"
                        maxStringContentLength="2147483647"
                        maxArrayLength="2147483647"
                        maxBytesPerRead="2147483647"
                        maxNameTableCharCount="2147483647" />

        </binding>

      </netNamedPipeBinding>

    </bindings>

  </system.serviceModel>

Answer 1

I think it likely that both of these intermittent blips in the timing of operations are by-products of the connection pooling mechanisms which both the named pipe and the TCP bindings use. The connection pools have a maximum idle time after which idle connections are removed from the pool. This creates an inherent race condition: occasionally an attempt to establish a WCF channel may be made on a connection which the other side has just discarded as idle.

I haven't tried it myself, but if you care more about the consistency of timings than the absolute times, you could try adjusting the connection pool settings on the binding's transport binding element, either to disable pooling (set MaxOutboundConnectionsPerEndpoint = 0) or to reduce the incidence of idle connections (change IdleTimeout values).

If you can't make that work, or if you believe the delays, when they occur, are longer than they ought to be even given the inherent variability which connection pooling introduces, you'll probably need help from Microsoft engineers, as this stuff is deep in the innards of the WCF implementation.