Crash when HTTP content length is very large

Question

I have a boost-beast async http client which crashes when the content length is very large. I have tested it with a ncat http server as

cat response.txt| sudo ncat -lvnp 443 --ssl

where response.txt contains my response. Curiously the code doesnt crash when the file is saved in an UNIX environment but crashes when the file is saved in Windows. The hexdump of the content length from the UNIX saved file is -1234 while the windows hex dump shows the original length. Contents of the file is

HTTP/1.1 200 OK
Connection: close
Cache-control: no-cache, no-store
Pragma: no-cache
X-Frame-Options: DENY
Content-Type: text/html; charset=utf-8
X-Content-Type-Options: nosniff
Content-Length: 214748364716

<html><body>
<h1>Hi</h1>
</body>
</html>

Following it the crash dump I have

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f63bcda5921 in __GI_abort () at abort.c:79
#2  0x00007f63bd798957 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f63bd79eae6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f63bd79eb21 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f63bd79ed54 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f63bd79f2dc in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x000000000047276a in __gnu_cxx::new_allocator<char>::allocate (__n=2, this=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/ext/new_allocator.h:111
#8  std::allocator_traits<std::allocator<char> >::allocate (__n=2, __a=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/alloc_traits.h:436
#9  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create (this=0x7f639c003be8, __old_capacity=<optimized out>, __capacity=<optimized out>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/basic_string.tcc:153
#10 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve (this=0x7f639c003be8, __res=<optimized out>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/basic_string.tcc:293
#11 0x00000000007613b7 in boost::beast::http::basic_string_body<char, std::char_traits<char>, std::allocator<char> >::reader::init (this=0x7f639c003c08, length=..., ec=...) at ../INPUT/boost/include/boost/beast/http/string_body.hpp:102
#12 boost::beast::http::parser<false, boost::beast::http::basic_string_body<char, std::char_traits<char>, std::allocator<char> >, std::allocator<char> >::on_body_init_impl (this=0x7f639c003b38, content_length=..., ec=...)
    at ../INPUT/boost/include/boost/beast/http/parser.hpp:448
#13 0x000000000076b130 in boost::beast::http::basic_parser<false>::put (this=0x7f639c003b38, buffer=..., ec=...) at ../INPUT/boost/include/boost/beast/http/impl/basic_parser.ipp:161
#14 0x000000000074bc4f in boost::beast::http::basic_parser<false>::put<boost::asio::mutable_buffer> (this=0x2, buffers=..., ec=...) at ../INPUT/boost/include/boost/beast/http/impl/basic_parser.hpp:41

Relevant code where it crashes

void HttpClientSsl::onWrite(const boost::beast::error_code& ec, std::size_t) {

    if (ec) {
        mResultCallback(ec, "on HttpClientSsl::onWrite", std::nullopt);
        return;
    }

    // make the mRes before reading
    // otherwise the operation behavior is undefined.
    mRes.emplace();
    mRes.value().body_limit(8 * 1024 * 1024);
    // Receive the HTTP response
    boost::beast::http::async_read(
        mStream, mBuffer, mRes.value(),
        [shared = shared_from_this()](auto ec, auto bytesTransferred) { shared->onRead(ec, bytesTransferred); });
}

I have the following definitions of the relevant vars

std::optional<boost::beast::http::response_parser<boost::beast::http::string_body>> mRes;
boost::beast::flat_buffer mBuffer;
boost::beast::ssl_stream<boost::beast::tcp_stream> mStream;

Am I reading it wrong that the content length is causing the crash?

The HTTP response shown is malformed. The body data is only 40-43 bytes in size (depending on linebreak formatting), nowhere near 214748364716 bytes. Besides, 214748364716 will overflow a 32bit integer, and although the HTTP protocol doesn't limit files to 2/4GB, it is very rare to transfer files of that size over HTTP. You might just be hitting a bug in the client's implementation. — Remy Lebeau, Jan 25 '22 at 23:50
Yes, the http response is intentionally malformed to stress test the code. My curiosity is whether something I am doing wrong in the https client code that I should fix. — Suman, Jan 26 '22 at 00:10
"*the http response is intentionally malformed to stress test the code*" - then congrats, it did its job. You broke boost's code. Now go file a bug report so they can fix it. — Remy Lebeau, Jan 26 '22 at 00:56

score 1 · Answer 1 · answered Jan 27 '22 at 13:14

I think you're reading it wrong, or this might have been a bug has been fixed in a more recent version of the library.

Here's a minimized reproducer that works fine here, whether body_limit is overridden or not:

#include <boost/beast.hpp>
#include <iostream>
namespace http = boost::beast::http;

static constexpr std::string_view example =
    "HTTP/1.1 200 OK\r\n"
    "Connection: close\r\n"
    "Content-Length: 214748364716\r\n"
    "\r\n"
    "<html><body>Hi</body></html>\r\n";

int main() {
    http::response_parser<http::string_body> p;

    boost::system::error_code ec;
    p.body_limit(8 * 1024 * 1024);
    p.put(boost::asio::buffer(example), ec);
    // if (!ec) p.put_eof(ec);

    std::cout << ec.message() << "\n";
}

See it with

boost 1.78.0 https://godbolt.org/z/zr4MMG749
Sadly there are currently problems using Wandbox/Compiler Explorer to demonstrate with older Boost versions (twitter)

_"Curiously the code doesnt crash when the file is saved in an UNIX environment"_ Are you sure you are using CR+LF newlines? See e.g. [`-C` option in the `ncat` manpage here](https://man7.org/linux/man-pages/man1/ncat.1.html#:~:text=Use%20CRLF%20for%20EOL%20sequencel) — sehe, Jan 27 '22 at 13:25
Upgraded to 1.78 and I dont think I cant reproduce it anymore. Thanks for the suggestions. — Suman, Jan 28 '22 at 21:57

Crash when HTTP content length is very large

1 Answers1