MonoDroid : Intermittent failure when reading a large json string from web service

Question

I use the code below on a thread (showing a progress dialog in ui) to read json string from a ASP.Net MVC web service. The data can be between 1 mb and 4 mb.

    public static class WebRequestEx
    {
        public static string ExecuteRequestReadToEnd(this WebRequest req)
        {
            var resp = req.GetResponse();
            using (var resps = resp.GetResponseStream())
            {
                StringBuilder sb = new StringBuilder();
                // read through the stream loading up the string builder
                using (var respRdr = new StreamReader(resps))
                {
                    //return respRdr.ReadToEnd();
                    while (!respRdr.EndOfStream)
                    {
                        sb.Append(respRdr.ReadLine());
                    }
                    return sb.ToString();
                }
            }
        }
}

The stacktrace is as follows.

I/mono    (15666): Stacktrace:
I/mono    (15666):
I/mono    (15666):   at System.Threading.WaitHandle.set_Handle (intptr) <0x0008b>
I/mono    (15666):   at System.Threading.EventWaitHandle..ctor (bool,System.Threading.EventResetMode) <0x00053>
I/mono    (15666):   at System.Threading.ManualResetEvent..ctor (bool) <0x0001f>
I/mono    (15666):   at (wrapper remoting-invoke-with-check) System.Threading.ManualResetEvent..ctor (bool) <0xffffffff>
I/mono    (15666):   at System.Net.WebAsyncResult.get_AsyncWaitHandle () <0x00073>
I/mono    (15666):   at System.Net.WebAsyncResult.WaitUntilComplete (int,bool) <0x00033>
I/mono    (15666):   at System.Net.WebConnectionStream.Read (byte[],int,int) <0x000b3>
I/mono    (15666):   at System.IO.StreamReader.ReadBuffer () <0x00047>
I/mono    (15666):   at System.IO.StreamReader.ReadLine () <0x0014b>
I/mono    (15666):   at System.WebRequestEx.ExecuteRequestReadToEnd (System.Net.WebRequest) <0x000ab>

The issue is that this does not happen every time and is intermittent. This is bad because it causes the whole app to freeze and progress dialog is stuck with the user unable to do anything with the app. I do have a try/catch block in the calling code but this exception seems to get around it.

I was using ReadToEnd earlier and that was blowing up intermittently as well so I switched to reading line by line.

The stack trace is not particularly helpful as somehting seems to be going on within Readline().

Thoughts/advice/alternatives?

Answer 1

Try to use this one:

    public static string ExecuteRequestReadToEnd(this WebRequest req)
    {
        var resp = req.GetResponse();
        using (var resps = resp.GetResponseStream())
        {
            var buffer = new byte[16 * 1024];
            using (var ms = new MemoryStream())
            {
                int read;
                while ((read = resps.Read(buffer, 0, buffer.Length)) > 0)
                {
                    ms.Write(buffer, 0, read);
                }
                var content = ms.ToArray();
                return Encoding.UTF8.GetString(content, 0, content.Length);
            }
        }
    }

Answer 2

You need to increase the maxArrayLength value because your byte[] array returned exceeds the 16348 size. The max value you can set for maxArrayLength is again the max Int32 number which is equal to 2147483647. Modify this setting and your web service will be able to transfer large data between your web service and the client application.

In Web.config , the web service reference bindings for clients are generated.

"The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element."

set it to the maximum of 2^31 - 1 or 2147483647 the biggest 32 bit number.

Pay attention on the transferMode, which by default is Buffered.

• Bufferred means the request and response messages will be both buffered.

Other enumerations for transferMode:

• Streamed which means both the request and response messages will be streamed, or
• StreamedRequest where the request will be streamed while response message buffered, or
• StreamedResponse where the request message will be buffered while response message streamed.

For those that don't know, Buffered means that the transfer will hold the entire message in memory buffer all until the transfer is completed. Streamed means that only the message headers will be buffered, while the message body will be exposed as a stream.

Change to the maxReceivedMessageSize from 65536 to 2147483647 - if you leave the transferMode="Buffered" we might get another error:

For TransferMode.Buffered, MaxReceivedMessageSize and MaxBufferSize must be the same value. Parameter name: bindingElement

Both attributes require the same value, the MaxReceivedMessageSize is larger than MaxBufferSize, if the whole message is buffered.

You have two options: a) Change the MaxBufferSize to size of 2147483647 b) Make the transferMode to: Streamed, StreamedRequest or StreamedResponse.

"The decision to use either buffered or streamed transfers is a local decision of the endpoint for HTTP transports. For HTTP transports, the transfer mode does not propagate across a connection, or to proxy servers or other intermediaries. Setting the transfer mode is not reflected in the description of the service contract. After generating a proxy to a service, you can (it is allowed but not required) edit the configuration file for services intended to be used with streamed transfers to set the transfer mode. For TCP and named pipe transports, the transfer mode is propagated as a policy assertion." http://msdn.microsoft.com/en-us/library/system.servicemodel.transfermode.aspx

Answer 3

Unless you have special requirements on your WebRequest you should use WebClient.

Methods like DownloadString and (even better) DownloadStringAsync will shield you from processing the response bits where it's easy to allocate way to much memory and temporary strings that will affect your application performance.