Problem with WCF dual binding callback channel: channel is faulty

Question

I am facing problem with client which cannot register callback channel on second start. I tried various things it seems I found a solution to remove Close() channel (ResetProxy() call in wrapper) call on shutdown of service. But this it turns out causing another problem: the server is crashing with error "The thread tried to read from or write to a virtual address for which it does not have the appropriate access". Additionally, there are network related issues which also result in same behavior. The solution is always to restart the server which is not really right option to fix callback channel registration. Could anyone suggest what could be the problem? I've done a lot of testing, and it seems this somehow relates to sudden stop of all clients (global stop of multiple services) and the number of services matters too. As result even new client cannot register with server until server is rebooted. This sounds like server is holding up closed channels.

Contract:

[ServiceContract(SessionMode = SessionMode.Required, CallbackContract = typeof(IAgentCallback))]
public interface IAgentManager
{
    [OperationContract]
    string RegisterAgent(string hostName);

    [OperationContract]
    bool Ping();
}

Client create callback code (truncated):

public class AgentCallbackDaemon
{
    private void CreateManagerProxy()
    {
        Reset();
        var isUnbound = true;
        while (isUnbound)
        {
            try
            {
                ManagerProxy = new ProxyWrapper<AgentManagerProxy, IAgentManager>(CreateProxy);
                ManagerProxy.ChannelStateChanged += HandleProxyConnectionStateChanged;
                isUnbound = false;
            }
            catch (AddressAlreadyInUseException)
            {
                sLog.ErrorFormat("Port is already reserved, binding failed.");
            }
            catch (Exception error)
            {
                sLog.ErrorFormat($"No proxy due to {error}");
                throw;
            }
        }
    }

    private AgentManagerProxy CreateProxy()
        => mCallbackChannelPortRange.IsCallbackPortRageSet() ? GetProxyWithPortRange() : GetProxyDefault();

    private AgentManagerProxy GetProxyDefault()
        => new AgentManagerProxy(mAgentCallback, mManagerUri, GetServiceName());

    private AgentManagerProxy GetProxyWithPortRange()
    {
        var minPort = mCallbackChannelPortRange.MinimumCallbackPortNumber;
        var maxPort = mCallbackChannelPortRange.MaximumCallbackPortNumber;
        return new AgentManagerProxy(mAgentCallback, mManagerUri, GetServiceName(), minPort, maxPort);
    }
}

Client callback code (truncated):

public class AgentManagerProxy : DuplexClientBase<IAgentManager>, IAgentManager
{
    public const string SERVICE_NAME = "AgentManager";

    public AgentManagerProxy(IAgentCallback callback, string serviceAddress, string connectionId,
        ushort minPort, ushort maxPort)
        : base(callback, BindingAbstractFactory.DuplexServiceClientBindingFactory(connectionId,
        minPort, maxPort), BindingUtility.CreateEndpoint(serviceAddress, SERVICE_NAME))
    {
    }

    public string RegisterAgent(string hostName)
    => Channel.RegisterAgent(hostName);


    public bool Ping() => return Channel.Ping();
}


public static class BindingAbstractFactory
{
    public static Binding DuplexServiceClientBindingFactory(string connectionId, ushort minPort, ushort maxPort)
    => CreateDuplexServiceClientBinding(connectionId, minPort, maxPort);


    private static Binding CreateDuplexServiceClientBinding(string connectionId, ushort minPort, ushort maxPort)
    {
        var binding = CreateDuplexServiceHostBinding();
        if (binding is WSDualHttpBinding)
        {
            lock (sLock)
            {
                try
                {
                    ((WSDualHttpBinding)binding).ClientBaseAddress =
                        CreateClientBaseAddress(connectionId, minPort, maxPort);
                }
                catch (Exception error)
                {
                    sLog.ErrorFormat("Unexpected exception: {0}", error);
                    throw error;
                }
                finally
                {
                    Monitor.PulseAll(sLock);
                }
            }
        }

        return binding;
    }

    private static Binding CreateDuplexServiceHostBinding()
    {
        var binding = new WSDualHttpBinding
        {
            ReceiveTimeout = TimeSpan.MaxValue,
            MaxReceivedMessageSize = int.MaxValue,
            ReaderQuotas =
            {
                // Set the maximum sizes to allow big message sizes.
                MaxArrayLength = int.MaxValue,
                MaxBytesPerRead = int.MaxValue,
                MaxDepth = int.MaxValue,
                MaxNameTableCharCount = int.MaxValue,
                MaxStringContentLength = int.MaxValue
            },
            Security =
            {
                // Disable security on control services.
                // TODO: Once security concept has been clarified, update this to reflect security rules.
                Mode = WSDualHttpSecurityMode.None,
                Message = { ClientCredentialType = MessageCredentialType.None }
                //// binding.Security.Mode = SecurityMode.None;
                //// binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.None;
            }
        };

        return binding;
    }

    private static Uri CreateClientBaseAddress(string connectionId, ushort minPort, ushort maxPort)
    {
        var fullDnsName = Dns.GetHostEntry(Dns.GetHostName()).HostName;
        int portNumber;
        if (maxPort == ushort.MaxValue)
            portNumber = LocalPorts.GetRandomAvailablePort(connectionId, minPort, maxPort);
        else
            portNumber = LocalPorts.GetNextAvailablePort(connectionId, minPort, maxPort);
        return new Uri($@"http://{fullDnsName}:{portNumber}");
    }
}

All this wrapped into proxy wrapper with Reset function which is called on service stop or any error (including network):

public class ProxyWrapper<TPROXY, TSERVICE> where TPROXY : ClientBase<TSERVICE> where TSERVICE : class
{
    public delegate TPROXY CreateProxy();
    private readonly object mProxyLock = new object();</summary>
    private readonly CreateProxy mCreateProxy;

    private TPROXY mProxy;

    public ProxyWrapper(CreateProxy createProxyCallback)
    {
        mLog.Info.Write($"Creating Proxy for '{typeof(TPROXY).FullName}'");
        mCreateProxy = createProxyCallback;
        BuildProxy();
    }

    public event EventHandler<CommunicationState> ChannelStateChanged;

    public TPROXY Proxy
    {
        get
        {
            lock (mProxyLock)
            {
                if (mProxy == null)
                    BuildProxy();

                return mProxy;
            }
        }
    }

    public void ResetProxy()
    {
        mLog.Info.Write("Call ResetProxy()");
        lock (mProxyLock)
        {
            try
            {
                if (mProxy == null)
                    return;

                UnSubscribeFromChannelEvents();
                CloseProxy();
            }
            catch (Exception ex)
            {
                // Catch all exceptions, and ignore them.
            }
            finally
            {
                mProxy = null;
            }
        }
    }

    private void RaiseChannelStateChanged(object sender, EventArgs e)
        => ChannelStateChanged?.Invoke(this, mProxy.InnerChannel.State);

    private void RaiseChannelEnteredFaultyState()
        => ChannelStateChanged?.Invoke(this, CommunicationState.Faulted);

    private void BuildProxy()
    {
        lock (mProxyLock)
        {
            if (mProxy != null)
                ResetProxy();

            mProxy = mCreateProxy();
            SubscribeToChannelEvents();
        }
    }

    private void SubscribeToChannelEvents()
    {
        if (mProxy?.InnerChannel == null)
            return;

        mProxy.InnerChannel.Faulted += OnInnerChannelFaulted;
        mProxy.InnerChannel.Closed += RaiseChannelStateChanged;
        mProxy.InnerChannel.Closing += RaiseChannelStateChanged;
        mProxy.InnerChannel.Opened += RaiseChannelStateChanged;
        mProxy.InnerChannel.Opening += RaiseChannelStateChanged;
    }

    private void UnSubscribeFromChannelEvents()
    {
        if (mProxy?.InnerChannel == null)
            return;

        mProxy.InnerChannel.Faulted -= OnInnerChannelFaulted;
        mProxy.InnerChannel.Closed -= RaiseChannelStateChanged;
        mProxy.InnerChannel.Closing -= RaiseChannelStateChanged;
        mProxy.InnerChannel.Opened -= RaiseChannelStateChanged;
        mProxy.InnerChannel.Opening -= RaiseChannelStateChanged;
    }

    private void OnInnerChannelFaulted(object sender, EventArgs e)
    {
        ResetProxy();
        RaiseChannelEnteredFaultyState();
    }

    private void CloseProxy()
    {
        try
        {
            mProxy.Close();
        }
        catch (Exception ex)
        {
            try
            {
                mProxy.Abort();
            }
            catch (Exception abortException) {  // ignored  }
        }

        try
        {
            mProxy.ChannelFactory.Close();
        }
        catch (Exception ex)
        {
            try
            {
                mProxy.ChannelFactory.Abort();
            }
            catch (Exception abortException) {  // ignored  }
        }
    }
}

The server only maintains the callback list.

Create Proxy for 'AgentManagerProxy'
Call BuildProxy()
mProxy.Open() [State=Created]
call SubscribeToChannelEvents()
...............
Service stopping...
call ResetProxy()
Call UnSubscribeFromChannelEvents()
call CloseProxy(): mProxy.Close()
call CloseProxy(): mProxy.ChannelFactory.Close()

Added logging for client shows this on first time start:

On second start service fails:

Create Proxy for 'AgentManagerProxy'
Call BuildProxy()
mProxy.Open() [State=Created]

Error: No proxy due to System.TimeoutException: The request channel timed out attempting to send after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://xxxxx:9003/AgentManager' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout. at System.ServiceModel.Channels.HttpChannelUtilities.SetRequestTimeout(HttpWebRequest request, TimeSpan timeout) at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) --- End of inner exception stack trace ---

score 0 · Answer 1 · answered Jun 10 '20 at 08:28

The number of your clients may exceed the value of Maxconcurrentinstances. Maxconcurrentinstances is a positive integer that limits the number of instancecontext objects that can be executed through ServiceHost at a time. When the number of slots is below the limit, requests to create other instances are queued and completed. The default value is the sum of maxconcurrentsessions and maxconcurrentcalls.You need to close the channel after each client-side completes.Otherwise, there will be an exception after the client has been queuing for a long time.

try
{
   if (client.State != System.ServiceModel.CommunicationState.Faulted)
   {
      client.Close();
   }
}
catch (Exception ex)
{
   client.Abort();
}

The close method may fail. If the close method fails, the abort method is required.

We have already implemented Throttling on server: Using MaxConcurrentCalls value: '144' Using IsPrioritizationEnabled value: 'False' Using MaxConcurrentInstances value: '5032' Using MaxConcurrentSessions value: '5000' Using PrioritySynchronizerThreadPoolSize value: '128' I run the test with 60 hosts on 20 different servers and this randomly failing. However, increasing this number to 200 (10 clients on each 20 servers) results in consistent failure results. — btomas, Jun 11 '20 at 13:24
Try adding the following configuration to the configuration file: — Ding Peng, Jun 18 '20 at 01:30

Problem with WCF dual binding callback channel: channel is faulty

1 Answers1