How to read the response stream before the Http response completes

Question

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

GetResponse() or the callback you provide for BeginGetResponse() are called as soon as all the response headers are read, but the entire response will not be read unless it is really small or you read it. — Gonzalo, May 17 '10 at 00:54
Not clear if the OP has indeed tested any of the suggested solutions and faced any specific problems. In my experience, get response stream only gets the stream and as you read from that stream, the response is downloaded, unless of course it is a small chunk! — Charles Prakash Dasari, May 21 '10 at 04:47

score 11 · Accepted Answer · edited May 23 '17 at 12:00

I think this is very close to what @Zachary suggests. And it (seems to) work(s); actually I think using using as @Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.

In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.

You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?

The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.

namespace StreamReadWebRequest
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Net;
    using System.IO;

    class Program
    {
        static void Main(string[] args)
        {
            HttpWebRequest req;
            HttpWebResponse res = null;

            try
            {
                req = (HttpWebRequest)WebRequest.Create(
                        "http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
                res = (HttpWebResponse)req.GetResponse();
                Stream stream = res.GetResponseStream();

                byte[] data = new byte[4096];
                int read;
                while ((read = stream.Read(data, 0, data.Length)) > 0)
                {
                    Process(data, read);
                }
            }
            finally
            {
                if (res != null)
                    res.Close();
            }
            Console.In.Read();
        }

        private static void Process(byte[] data, int read)
        {
            Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
        }
    }
}

score 3 · Answer 2 · edited Jul 23 '14 at 13:54

I was looking for the same thing: server streams chunked XML data and I needed a C# client that could access this data while server is streaming. I tried many different ways to access the source (WebChannelFactory, WebClient, HttpWebRequest/Response, TcpClient) but failed so far. Finding this thread I focused on HttpWebRequest/Response where I have the same problem that following line is blocking:

HttpWebResponse resp = (HttpWebResponse)request.GetResponse();

As Artiom Chilaru stated, if it's blocking: something is wrong, because it should not. Now focusing on trying to replicate default behavior with downloading large .ISO files I found out that Fiddler was blocking the GetResponse() method!

However there is no problem to open Fiddler once the stream has been set up (i.e. GetResponse() has already been called), but during the HTTP GET if you find GetResponse() is blocking try to close Fiddler and see if your application now continuous it's normal flow (i.e. reading the stream).

This was exactly my issue. Once I closed Fiddler GetResponse() behaved as expected. Thanks! — Ray Ackley, Feb 25 '15 at 19:52
I know this is an old thread, but adding my 2 cents. Fiddler is actually a proxy server, so it actually gets the entire response from the original server and then does what you are expecting it to do. That's why its blocking. — M22an, Dec 06 '18 at 12:12

score 1 · Answer 3 · answered May 16 '10 at 05:43

If you set the buffer size on your read, you can read in the data in chunks... example...

 // Get the response stream
 using(Stream resStream = response.GetResponseStream())
 {

        string parseString = null;
        int    count      = 0;

        do
        {
            // Read a chunk of data
            count = resStream.Read(buf, 0, buf.Length);

            if (count != 0)
            {
                // Convert to ASCII
                parseString = Encoding.ASCII.GetString(buf, 0, count);

                // Append string to results
                sb.Append(tempString);
            }
        }
        while (count > 0);

}

This is after the response is complete, I want to be able to read the response stream before that. — , May 16 '10 at 11:22

score 1 · Answer 4 · edited May 23 '17 at 12:19

I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.

After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.

If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?

Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.

How to read the response stream before the Http response completes

4 Answers4

Linked