Asynchronous WCF calls timeout on client side after 10 seconds

Question

EDIT

After looking at this for a while, I thought it might be a configuration issue on my development box. However, after doing a clean download of the source code to a different development machine, I'm still getting this issue.

I've got a Silverlight client that calls WCF services asynchronously. Intermittently, I will get one of the generic NotFound exceptions. The exceptions (which notoriously lack detail) happen intermittently on pretty much any of the services that I call.

Here's the thing. With judicious setting of breakpoints, I've been able to determine that the service-side is executing normally. Data is being retrieved and returned. The problem seems to be more on the client side of things.

Here's the rub... I can consistently make the exception happen if I make the service execute for more than 10 seconds. When I do, it never makes it back to my completed callback. Instead, I get the exception in the client-side Reference.cs for the service:

        public System.Collections.ObjectModel.ObservableCollection<Project.Ui.SilverLight.ServiceName.ModelName> EndGetService(System.IAsyncResult result) {
            object[] _args = new object[0];
            System.Collections.ObjectModel.ObservableCollection<roject.Ui.SilverLight.ServiceName.ModelName> _result = ((System.Collections.ObjectModel.ObservableCollection<roject.Ui.SilverLight.ServiceName.ModelName>)(base.EndInvoke("GetService", _args, result)));
            return _result;
        }

The exception I get is (not very helpful):

System.ServiceModel.CommunicationException was unhandled by user code
  Message=The remote server returned an error: NotFound.
  StackTrace:
       at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
       at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result)
       at Project.Ui.SilverLight.Service.ServicesClient.ServicesClientChannel.EndGetxxxxx(IAsyncResult result)
       at Project.Ui.SilverLight.Service.ServicesClient.Project.Ui.SilverLight.Service.IServices.EndGetxxxx(IAsyncResult result)
       at Project.Ui.SilverLight.Service.ServicesClient.OnEndGet(IAsyncResult result)
       at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result)
  InnerException: System.Net.WebException
       Message=The remote server returned an error: NotFound.
       StackTrace:
            at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
            at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
            at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
       InnerException: System.Net.WebException
            Message=The remote server returned an error: NotFound.
            StackTrace:
                 at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
                 at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClass5.<EndGetResponse>b__4(Object sendState)
                 at System.Net.Browser.AsyncHelper.<>c__DisplayClass2.<BeginOnUI>b__0(Object sendState)
            InnerException:

Binding info (names changed, but they do match the services being executed)

    <binding name="Project.WebUI.Services.xxxxxServices.customBinding0" closeTimeout="00:01:00"
      openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00">
      <binaryMessageEncoding/>
      <httpTransport />
    </binding>

...

  <service name="Project.WebUI.Services.xxxxxServices">
    <endpoint address="" binding="customBinding" bindingConfiguration="Project.WebUI.Services.xxxxxServices.customBinding0"
      contract="Project.WebUI.Services.xxxxxServices" />
    <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
  </service>

I believe I've checked the relevant timeouts. The channel's operationTimeout is set to at least one minute, as is the ReceiveTimeout and OpenTimeout. Is there something arcane about Silverlight async WCF calls that needs to be set to tell it to go more than ten seconds?

Why do you call that a timeout? It looks a lot like a `NotFound` error (404). — John Saunders, Sep 17 '10 at 20:42
But it's not "not found." It really is a timeout that's happening here. If the service runs less than 10 seconds, I'm golden. That's what's driving me crazy. I know the service is executing, and I know that it will return valid data. As long as it does it in less than 10 seconds, no error. — Robaticus, Sep 18 '10 at 00:49
@Robaticus: I doubt that .NET imagined seeing a 404 error somewhere. — John Saunders, Sep 18 '10 at 01:00
@John - I hear you, but I also see what's happening with my own two eyes. :) I did read somewhere that sometimes Silverlight will mask a fault as a NotFound. Anyway, it's really looking like a machine configuration problem of some sort. Associate tried the same codebase, and it worked fine. I may have to switch to coding on the laptop for a while. — Robaticus, Sep 18 '10 at 01:30
@Robaticus: I suggest you ask Fiddler what's happening, or turn on WCF tracing, both on server, and on client if possible. Maybe it's the other way around - SilverLight hiding the 404 error, therefore timing out when no non-error response was received. — John Saunders, Sep 18 '10 at 04:13
@John I'll give that a try again. I have some "fiddling" to do with fiddler to get it to work with the local addresses without cross-domain issues cropping up. — Robaticus, Sep 18 '10 at 04:18
The 404 Not Found error is an artifact of how Silverlight handles its WCF calls: it's what gets returned to Silverlight by the HTTP stack whenever the HTTP call fails for some reason. It doesn't have anything to do with the actual, underlying error. — Ken Smith, Sep 23 '10 at 15:36
Well, I've finally gotten Fiddler2 to play nicely. It's definitely a client-side issue, as the service is completing successfully, and returning a 200. However, even before the 200 is returned, I'm getting the client-side "NotFound" exception in the EndInvoke for the call. — Robaticus, Oct 06 '10 at 12:48
@Robaticus: Did you ever get this fixed? If yes, how? We've been experiencing the same problem for a while now, totally random NotFound errors, FaultBehavior not helping, etc. My gut tells me it has something to do with a timeout as well, or at least that's how it feels... — efdee, Jan 25 '11 at 12:05
@efdee - Never did get it worked out. I wound up switching to using FireFox for development purposes. I think it has something to do with beta-bits of Silverlight that never got uninstalled properly. — Robaticus, Jan 25 '11 at 13:35
@Robaticus: Switching to Fx solved it for you? Weird. I should try that. I can confirm that (at least here) the problem is not related to beta leftovers, I never had a beta for anything on this machine. Are you using SL3 or SL4 ? — efdee, Jan 26 '11 at 08:02
@efdee - yep, using FireFox for debugging solved the problems. I tried it with both IE8 and IE9 (beta) and still had the timeout issues. I never went through the hassle of getting Chrome to talk to the debugger. Whatever it is affects both my main dev system and my laptop, so it has to be associated with *something* that I have installed. I just don't know what that *something* is. — Robaticus, Jan 26 '11 at 13:07

John Nicholas · Answer 1 · 2010-10-04T11:52:46.207

The reason why you are gettign not found is NOT due to a time out. I lost a week trying to figure this out.

I have references to silverlight but this is not a silverlight specific problem.

The not found fault exists because of the way the error codes are produced. You need to intercept any faultsand change the error code - this will cause the errors to be reported properly and not fuiltered out due to security.

If you think about it, the server is not found because the call is asynchronous and it has died due to an exception in your code.

I use this class in my wcf service to plug in - i am doing this through the web.config, however its not hard to change this so that its done wit ha class level attribute on the service itself.

you will probably want to strip out all the Contract stuff. Its being used in the project i am working on but is not related to this problem in any way.

 [AttributeUsage(AttributeTargets.Class)]
    public class FaultBehavior : Attribute, IServiceBehavior,  IEndpointBehavior
    {  

    public class SilverlightFaultMessageInspector : IDispatchMessageInspector
    {

        public void BeforeSendReply(ref Message reply, object correlationState)
        {
            Contract.Assume(reply !=null );
            if (reply.IsFault)
            {
                HttpResponseMessageProperty property = new HttpResponseMessageProperty();

                // Here the response code is changed to 200.
                property.StatusCode = System.Net.HttpStatusCode.OK;

                Contract.Assume(reply.Properties != null);
                reply.Properties[HttpResponseMessageProperty.Name] = property;
            }
        }

        public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
        {
            // Do nothing to the incoming message.
            return null;
        }


    }


    #region IServiceBehavior Members

    void IServiceBehavior.AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters)
    {

    }

    void IServiceBehavior.ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
    {
        Contract.Assume( serviceHostBase != null);
        Contract.Assume(serviceHostBase.ChannelDispatchers != null);

        foreach (ChannelDispatcher cDispatcher in serviceHostBase.ChannelDispatchers)
        {
            Contract.Assume(cDispatcher != null);
            Contract.Assume(cDispatcher.Endpoints != null );
            foreach (EndpointDispatcher endpointDisbatcher in cDispatcher.Endpoints)
            {
                Contract.Assume(endpointDisbatcher != null);
                Contract.Assume(endpointDisbatcher.DispatchRuntime  != null);
                Contract.Assume(endpointDisbatcher.DispatchRuntime.MessageInspectors  != null);
                endpointDisbatcher.DispatchRuntime.MessageInspectors.Add(new SilverlightFaultMessageInspector());
            }
        }
    }

    void IServiceBehavior.Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
    {

    }

    #endregion

    #region IEndpointBehavior Members

    void IEndpointBehavior.AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
    {

    }

    void IEndpointBehavior.ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
    {

    }

    void IEndpointBehavior.ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
    {
        Contract.Assume(endpointDispatcher != null);
        Contract.Assume(endpointDispatcher.DispatchRuntime != null);
        Contract.Assume(endpointDispatcher.DispatchRuntime.MessageInspectors != null);
        SilverlightFaultMessageInspector inspector = new SilverlightFaultMessageInspector();
        endpointDispatcher.DispatchRuntime.MessageInspectors.Add(inspector);
    }

    void IEndpointBehavior.Validate(ServiceEndpoint endpoint)
    {

    }

    #endregion
}

you then need to define the element that will appear in the web.config liek this

public class FaultHandlerElement   : BehaviorExtensionElement 
{
    protected override object CreateBehavior()
    {
        return new FaultBehavior();
    }

    public override Type BehaviorType
    {
        get { return typeof(FaultBehavior); }
    }


}

and then in the web config you need to add this to the service model section - fault exception will have a wavy line under it ;)

<system.serviceModel>
    <extensions>
        <behaviorExtensions>
            <add name="faultBehaviourExtension"
  type="CopSilverlight.Web.FaultHandlerElement, CopSilverlight.Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/>
        </behaviorExtensions>
    </extensions

you then hook your behaviours up to use this like so

   <serviceBehaviors>
                <behavior name="basicHttpBehaviour">
                    <serviceMetadata httpGetEnabled="true" />
                    <serviceDebug includeExceptionDetailInFaults="true" />
                    <faultBehaviourExtension />
                </behavior>

An alternative 'fix' is to turn on tracing in your wcf services oen thing to note is that if you do not make the directory it will simply not produce a log

    <system.diagnostics>
        <sources>
            <source name="System.ServiceModel"
                switchValue="Information, ActivityTracing"
                propagateActivity="true">
                <listeners>
                    <add name="traceListener"
  type="System.Diagnostics.XmlWriterTraceListener"
  initializeData="c:\logs\appName\wcf.svclog"  />
                </listeners>
            </source>
        </sources>
    </system.diagnostics>

the logging is no way ideal though - the first solution actually fixes the problem.

I have implemented both of these and now i get exceptions bubbling up to my silverlight app as you would expect.

I'm still thinking that there is an element of timing here, but I believe your answer will help me track down where it's happening. thanks. — Robaticus, Oct 04 '10 at 12:56
well thre may well be. However the Not Found error is simply a facade that is stopping you from seeing the real cause. — John Nicholas, Oct 04 '10 at 13:42
With additional testing, I've found that it is definitely something that happens client-side. — Robaticus, Oct 06 '10 at 12:50
ar eyou getting the not found exception after what i suggested? Because your problem was this exception. The not found is being caused by the wcf service setting the fault error code to 500. If you have changed this to 200 then you will see different exceptions which will allow you to fix your code. If the problem was on the client then why would the wcf service be giving you an error? Yes you may be passing bad data into it, but the fact remains to get at the real cause you need to use what i suggested above in order to get at the real exception and stack trace. I answered your question. — John Nicholas, Oct 06 '10 at 13:01
And yes, the exception will always appear in reference.cs despite origionating in the wcf. — John Nicholas, Oct 06 '10 at 13:18
How are you setting breakpoints and hitting them in an asynchronous service call? To do this i have to set up a synchronous client - which causes this behaviour to disapear. — John Nicholas, Oct 06 '10 at 13:21

score 3 · Answer 2 · answered Sep 30 '10 at 12:51

3

Could it be something on the network.

In this they got the same error message but it was really a 504 Gateway timeout.

If for example you have a proxy server between the silverlight app and the server, that closes the connection so that the callback cannot reach the client.

answered Sep 30 '10 at 12:51

Shiraz Bhaiji

64,065
34
143
252

Could you check what the proxy settings are on your machine, it is possible that even for a local address it is going to the proxy server and back again – Shiraz Bhaiji Oct 04 '10 at 10:36

score 1 · Answer 3 · answered Sep 17 '10 at 20:40

1

In the config files above, it looks like they're from the server's configuration. Have you checked the client-side configuration as well? The SendTimeout and ReceiveTimeout properties on the client channel both default to one minute, but it's possible they could have been adjusted or overridden. And of course, you should check both your configuration XML files, but also where you create the binding in code, as it's possible to set properties there which conflict with your configuration files. (On both the client and on the server.)

answered Sep 17 '10 at 20:40

Ken Smith

20,305
15
100
147

I've checked both, and even specifically set up the client to use more than 10 seconds for everything. I'm still seeing this issue. I'm really starting to think that my development server simply has some infrastructure piece (Silverlight SDK, WCF, something?) that's out of date. – Robaticus Sep 23 '10 at 15:21
That's certainly possible -- I've had that happen to me before. But I'd start next by configuring WCF tracing (as described, say, here: http://software.intel.com/en-us/blogs/2010/02/03/adventures-with-silverlight-and-wcf-the-remote-server-returned-an-error-notfound/). You should probably also enable Silverlight faults on your WCF service, as described here: http://msdn.microsoft.com/en-us/library/dd470096(VS.96).aspx. That may provide Silverlight a bit more information about what's going wrong. – Ken Smith Sep 23 '10 at 15:40
Thanks. Though I think I did that, I'll verify tonight and reimplement Silverlight faults. If I weren't already bald, I think I would be from pulling my hair out on this one. – Robaticus Sep 23 '10 at 16:49

Aliostad · Answer 4 · 2010-10-04T07:49:00.777

0

Just out of curiosity, what is the idleTimeout in web.config set at the moment? Is it possibly set to 10 minutes??

UPDATE

So here is your problem! This is a known issue with idleTimeout! Just remove that setting and things should work again.

edited Oct 04 '10 at 07:49

answered Oct 03 '10 at 22:25

Aliostad

80,612
21
160
208

I believe it is set to 00:10:00 – Robaticus Oct 03 '10 at 23:24
Looked, and, unfortunately, that wasn't it. I've explicitly set the timeouts for all four of the timeouts to be more than a minute. – Robaticus Oct 06 '10 at 12:49

Asynchronous WCF calls timeout on client side after 10 seconds

4 Answers4