c# WebResponse content length limit

Question

I'm trying to fetch a big file from internet with C# with WebRequest. So far all other stuff that I have is working fine, but it appears that some part in the middle of the response I get is removed. I tried printing out the response.ContentLength and getResponseContent(response).Length and I got -1 and 80000 respectively. Is there a limit on the WebResponse length or it's the helper function that's buggy? How do I get the full content?

Here's my getResponseContent function:

    private static String getResponseContent(HttpWebResponse response)
    {
        Stream responseStream = response.GetResponseStream();

        byte[] buffer = new byte[1000];
        String ret = "";

        while (responseStream.Read(buffer, 0, 1000) > 0)
            ret += (System.Text.Encoding.Default.GetString(buffer));

        return ret;
    }

Thanks to @Neolisk, I've re-written my getResponseContent to use the StreamReader class and it works like magic. Here's the code:

    private static String getResponseContent(HttpWebResponse response)
    {
        Stream responseStream = response.GetResponseStream();
        StreamReader sr = new StreamReader(responseStream);
        return sr.ReadToEnd();
    }

However, still, can anyone explain why the the ContentLength in the response header is -1 instead some meaningful length?

Unrelated to your question, but consider using a string builder. `+=` append is memory inefficient, because it creates a new string every time. — Victor Zakharov, Jul 14 '14 at 15:58
Because you are dealing with bytes -> String, you can use [StreamReader](http://msdn.microsoft.com/en-us/library/system.io.streamreader(v=vs.110).aspx) class, should be easier a bit, using [ReadToEnd](http://msdn.microsoft.com/en-us/library/system.io.streamreader.readtoend(v=vs.110).aspx). — Victor Zakharov, Jul 14 '14 at 16:02

usr · Answer 1 · 2014-07-14T16:13:35.027

1

You are not using the return value from responseStream.Read to find out how many bytes were received.

edited Jul 14 '14 at 16:13

answered Jul 14 '14 at 15:57

usr

168,620
35
240
369

`Read(...) > 0` ? http://msdn.microsoft.com/en-us/library/system.io.stream.read(v=vs.110).aspx `zero (0) if the end of the stream has been reached` – Victor Zakharov Jul 14 '14 at 16:00
@usr I was referring to the helper function that I wrote when I said it's buggy... but thanks anyway – Tengyu Liu Jul 14 '14 at 16:05
@Neolisk yeah, he terminates reading correctly but he does not check how many bytes were read. – usr Jul 14 '14 at 16:14

score 1 · Accepted Answer · answered Jul 14 '14 at 16:03

Without knowing what the actual data you're receiving is, it's hard to be sure what's going on.

However - response.ContentLength is the header value from the HTTP response, which is the length of the response in bytes.

How many characters in a string that represents is not necessarily the same thing - many unicode type encodings are more than one byte per character.

Use the responseStream.Read overload that puts binary data into a byte[] buffer, create a decoder based on what the actual encoding is (set in the header) and decode your string from there.

But why does the header has -1 instead of some meaningful length? Could it be a server-side issue? — Tengyu Liu, Jul 14 '14 at 16:13

score 1 · Answer 3 · answered Jul 14 '14 at 16:15

1

The ContentLength is a http header. If the server side didn't send it you'll receive -1. It's not mandatory.

That's why using the ResponseStream is a good idea.

answered Jul 14 '14 at 16:15

Eric Lemes

561
3
10

c# WebResponse content length limit

3 Answers3