3

I'm attempting to get the content-length of a remote file so that I can limit the download size of remote images thus:

 WebResponse response = this.GetWebRequest().GetResponse();

 long contentLength = response.ContentLength;

 // WebResponse.ContentLength doesn't always know the value, 
 // it returns -1 in this case.
 if (contentLength == -1)
 {
     // Response headers may still have the Content-Length inside of it.
     string headerContentLength = response.Headers["Content-Length"];

     if (!String.IsNullOrWhiteSpace(headerContentLength))
     {
         contentLength = long.Parse(headerContentLength,
                                            CultureInfo.InvariantCulture);
     }
 }

My problem is that since the content-length header is optional, sometimes that value is indeterminable.

I have read in a similar questions here and here that seems to indicate that I could somehow use the Transfer-Encoding=chunked property to determine the length but I have found no example of how to do so.

Is this possible? Can anyone provide me with a code example?

Community
  • 1
  • 1
James South
  • 10,147
  • 4
  • 59
  • 115
  • 2
    It is optional (although I would assume will be almost always there for images), so no you cannot determine this value in all cases. – BrokenGlass Jan 12 '12 at 14:12
  • @BrokenGlass I would have thought so but both debugging and checking with firebug indicate otherwise. The image I'm looking at specifically is here http://images.mymovies.net/images/film/cin/500x377/fid10122.jpg – James South Jan 12 '12 at 14:17
  • @James South: I see a `Content-Length: 33566` in the response for that specific URL: http://hurl.it/hurls/c79a3f4611819c370e53a30d07958ee1892862b9/ce4aba5e6c9029ed454632536404f2496544c6b0 – Piskvor left the building Jan 12 '12 at 14:25
  • @Piskvor: Bizarre... hurl.it must be downloading file and reporting the length or using something smarter than my development server as I am not getting that property at all. – James South Jan 12 '12 at 14:37
  • @James South: All my clients (FF 9, Chrome 16, IE 6,7,8, Opera 11, wget 1.12) seem to be in agreement with Hurl here (from multiple locations in the Internet, no less) - I'm seeing the exact same headers. Are you sure it's not something on your side? – Piskvor left the building Jan 12 '12 at 14:48
  • @Piskvor: Am I being stupid? https://skydrive.live.com/?cid=46b6ee8fd7feac15&id=46B6EE8FD7FEAC15!242&sff=1 – James South Jan 12 '12 at 15:03
  • @James South: I never said anything like that; however, there might be something between you and the origin server (e.g. a proxy) modifying your HTTP traffic. – Piskvor left the building Jan 12 '12 at 15:06
  • Hmm...I don't see `Content-Length` in Firebug either, but I see it pass across the wire when I capture the raw network traffic with Wireshark! So this may be a Firebug's display bug; perhaps try looking at the traffic in Fiddler2, it's a great HTTP debugging proxy (for Windows). Firefox's add-ins perhaps don't get the full headers (?); as Fiddler is a proxy, it shows you exactly what goes across the wire. – Piskvor left the building Jan 12 '12 at 15:10
  • @Piskvor: Cheers.I'm thinking now it might just be easier to create a site white-list and allow remote files from there only. – James South Jan 12 '12 at 15:14

1 Answers1

3

No, this is impossible in the general case.

The only way to be sure is to download the whole resource, and see how big it was. Especially with dynamically generated and/or streamed resources, it may not be possible to tell what size the response will be (e.g. a live stream doesn't necessarily even have a well-defined beginning, or an end - it can just keep on sending the stream as one huge chunked response).

The chunked property is a workaround for an unknown size - the chunks are preceded with the size of the following chunk; however, there's no way to predict how many chunks there will be, nor what size.

Also: caveat Content-Length, there are (even in 2012!) buggy implementations which will misreport it.

Piskvor left the building
  • 91,498
  • 46
  • 177
  • 222
  • So there is no way to open, download to an upper limit then close a chunked connection once you have opened it? – James South Jan 12 '12 at 14:19
  • @James South: Oh, sure - you can open, send a request, and then keep downloading until the server tells you "no more chunks". There's just no way to tell *when exactly* that will happen, *until it happens* (at which point you have the complete response, and you can just check its byte size). – Piskvor left the building Jan 12 '12 at 14:22
  • So what I need to do is to download each chunk at a time checking the length of the next one and adding the respective chunk lengths closing the connection once the combined length hits my upper bite size limit? – James South Jan 12 '12 at 14:30
  • @James South: You may get an incomplete response that way (if your upper limit is too low); a better approach would be to close the connection when you get a zero-sized chunk (as that's the "transmission complete" marker). But yes, setting a limit may be useful, if you have a rough idea of the size you're expecting (e.g. 5 MB for images should be mostly sufficient, in 2012). – Piskvor left the building Jan 12 '12 at 14:33
  • Cool... I would know roughly what my upper limit would be so any idea how to code that then? I'm stumped. – James South Jan 12 '12 at 14:39
  • @James South: You may want to look around the Net (someone might have done that for you already), and then perhaps ask it as a new question. (See the RFC for specification, it's actually quite simple: http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1 ) – Piskvor left the building Jan 12 '12 at 14:53