Why isn't reqwest returning the content length in this request?

Question

I'm confused why I'm not getting any content back from the following function, which uses reqwest:

  fn try_get() {
      let wc = reqwest::Client::new();
      wc.get("https://httpbin.org/json").send().map(|res| {
          println!("{:?}", res);
          println!("length {:?}", res.content_length());
      });
  }

I'm expecting this function to display the response object and then give me the content length. It does the first but not the second:

Response { url: "https://httpbin.org/json", status: 200, headers: {"access-control-allow-credentials": "true", "access-control-allow-origin": "*", "connection": "keep-alive", "content-type": "application/json", "date": "Tue, 26 Feb 2019 00:52:47 GMT", "server": "nginx"} }
length None

This is confusing because if I hit the same endpoint using cURL, it gives me a body as expected:

$ curl -i https://httpbin.org/json
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Tue, 26 Feb 2019 00:54:57 GMT
Server: nginx
Content-Length: 429
Connection: keep-alive

{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}

What is the problem with my function that it does not provide me with the content length?

Answer 1

The reqwest documentation for content_length() is always a good place to start. It states

Get the content-length of the response, if it is known.

Reasons it may not be known:

• The server didn't send a content-length header.
• The response is gzipped and automatically decoded (thus changing the actual decoded length).

Looking at your example curl output, it contains Content-Length: 429 so the first case is covered. So now lets try disabling gzip:

let client = reqwest::Client::builder()
  .gzip(false)
  .build()
  .unwrap();

client.get("https://httpbin.org/json").send().map(|res| {
  println!("{:?}", res);
  println!("length {:?}", res.content_length());
});

which logs

length Some(429)

so the second case is the issue. By default, reqwest appears to be automatically handling gzipped content, whereas curl is not.

The Content-Length HTTP header is entirely optional, so generally relying on its presence would be a mistake. You should read the data from the request using the other reqwest APIs and then calculate the length of the data itself. For instance, you might use .text()

let wc = reqwest::Client::new();
let mut response = wc.get("https://httpbin.org/json").send().unwrap();
let text = response.text().unwrap();

println!("text: {} => {}", text.len(), text);

Similarly, for binary data you can use .copy_to():

let wc = reqwest::Client::new();
let mut response = wc.get("https://httpbin.org/json").send().unwrap();

let mut data = vec![];
response.copy_to(&mut data).unwrap();

println!("data: {}", data.len());