2

I have a method which is intended to download a file from an HTTP URL to a byte array:

private static byte[] DownloadFileToByteArrayWorker(HttpWebRequest Request, int bufferLength)
{
    byte[] responseByes = null;

    //Round up to the nearest multiple of 1024
    bufferLength = AdjustBufferLength(bufferLength);

    Request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
    Request.ServicePoint.Expect100Continue = false;
    Request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");

    using (var Response = (HttpWebResponse)Request.GetResponse())
    {
        using (Stream ResponseStream = Response.GetResponseStream())
        {
            using (MemoryStream ms = new MemoryStream())
            {
                int count = 0;
                byte[] buf = new byte[bufferLength];
                while ((count = ResponseStream.Read(buf, 0, buf.Length)) > 0)
                {
                    ms.Write(buf, 0, count);
                }
                responseByes = ms.ToArray();
            }
        }
    }

    return responseByes;
}

Request.GetResponse() is throwing a time out exception no matter how long I make the Timeout property of the HttpWebRequest. I can verify via my logs that the program is waiting the full Timeout period before erroring out, however, correlating my logs with the web server logs indicates that the web server is sending back a response almost immediately.

An interesting note is that when I access the same web server via the load balancer rather than directly, it downloads the file practically instantly. Also, if I access the URL via the web server directly in a web browser (no proxy needed, btw) I can download the file from individual web servers instantly that way too.

Some additional details:

  • I am using .NET Framework 4.7 on Windows 2012 R2.
  • The web server I'm trying to connect to is Apache on RHEL7. I'm not sure about the specific Apache version
  • I am connecting to the web server on a specific port which is reserved for HTTP traffic (a separate website is hosted on a different port number for HTTPS)
  • There's no web proxy

Any suggestions?

Mike Bruno
  • 600
  • 2
  • 9
  • 26
  • 2
    `while (ResponseStream.CanRead ... `? Try something like: `while ((count = responseStream.Read(buf, 0, buf.Length)) > 0) { }`. Move `byte[] buf = new byte[1024];` outside the loop. `1024` is low. `8192` is better. – Jimi Sep 08 '20 at 14:36
  • @Jimi Thanks for the advice. I did incorporate both of your suggestions (the while loop and increasing the buffer length), but its still timing out in the exact same way. Very odd. – Mike Bruno Sep 08 '20 at 19:06
  • 1
    Well, I cannot see the modifications you applied. If the Server you're connecting to is an HTTPS Server, it *may* enable HSTS. If this protocol is active, the Server *awaits* a response from you. Until the connection times out. The usual solution is to specify a User-Agent header using the header of a WebBrowser that doesn't support it (as IE11). But it's just theory. Add some more details, as the .Net Framework in use, the presence of Proxies, the result of `tracert.exe` when querying the Server by name etc. – Jimi Sep 08 '20 at 19:29
  • 1
    Btw, you should always use a HttpWebRequest, not a WebRequest object (returned by `WebRequest.CreateHttp()`) and cast `Request.GetResponse()` to `HttpWebResponse`: these classes are not the same thing. If this code is run from Windows 7 or Windows Server 2008 (or even later sometimes, depending on some witty configuration), add `ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;` before you create any connection (assuming it's a HTTPS Server which supports the protocol). – Jimi Sep 08 '20 at 19:36
  • 1
    Also, add `[HttpWebRequest].AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate; [HttpWebRequest].ServicePoint.Expect100Continue = false; [HttpWebRequest].Headers.Add(HttpRequestHeader.CacheControl, "no-cache");` – Jimi Sep 08 '20 at 19:37
  • 1
    Pass to the method a HttpWebRequest (as described). Everything else is related to this object, except `ServicePointManager.SecurityProtocol`, which must be set before the WebRequest creates a `HttpWebRequest` object. Note that, given the problem you're reporting, I doubt this is important. But set it anyway (if the Server uses that protocol - or TLS1.1, that is). In this cases, you try to make the Server return a distinct exception (btw, using HttpWebRequest, you should alwats do everything in a `try/catch` block: this is an old class, this is its MO). – Jimi Sep 08 '20 at 19:48
  • @Jimi Thanks, I was confused by the brackets initially. Do you want to submit an answer? I'd like to give you the credit if your suggestions solve my issue! – Mike Bruno Sep 08 '20 at 19:53
  • These are just *blind* suggestions, I'm not really sure this can solve your problem. If you get it to work, I suggest you post the solution yourself, so you can describe the original problem, which is more useful for others. – Jimi Sep 08 '20 at 19:56
  • Put 1 step before call `DownloadFileToByteArrayWorker` it looks there is some problems before calling it. – A Farmanbar Sep 17 '20 at 00:02
  • Not related to your problem - you can use `[Stream].CopyTo` instead of your loop. – Mike Sep 17 '20 at 07:29
  • Is there a redirect (302) involved? – Mike Sep 17 '20 at 07:30
  • Can you share a fiddler network capture? – Ygalbel Sep 17 '20 at 12:16

1 Answers1

0

As you said your code has problem only when you call the load balancer, I think the problem is the your client send a 100 continue request but your load balancer don't know how to handle it.

That is the reason you client doesn't send all the data right after the beginning of connection.

You can find more information about 100 continue in HTTP rfc section 8.2.3.

To fix the behavior from client side in c# you have to add this code:

ServicePointManager.Expect100Continue = false;

You can see the full documentation about this feature here.

Ygalbel
  • 5,214
  • 1
  • 24
  • 32