0

I have implemented a proxy with the boost beast library, where I have used the async https server and client examples from boost organization. In my proxy I was using the http::request<http::string_body> and http::response<http::string_body> types of messages, as used in the examples.

This proxy works well, except that it cannot receive (download) big files and streams.

So for that purpose, I decided to rework my transmitting mechanism by combining two examples from the https://www.boost.org/doc/libs/1_66_0/libs/beast/example/doc/http_examples.hpp. The mentioned examples are the "Example: Incremental Read" and the "Example: Send Child Process Output".

This example (that follows) partly works, but it has some issues.

Very often, when I have a series of requests to execute on one connection, I failed to read the second or the third response (the header) to a successfully written request, and therefore the connection gets broken, and the client (browser) reconnects and tries to execute them in a different session. This makes traffic very slow and annoying.

Although the proxy app is written as async one, this method is written in a synced (blocking) manner and serves only to receive the response from the host (upstream) in chunks and write the received data chunks directly as they are received to the original client (downstream).

The question is what is it that I am doing wrong?

I believe an experienced beast boost user could pick up the problem from reading the example.

std::shared_ptr<beast::ssl_stream<beast::tcp_stream>> m_sslDownStream;
std::shared_ptr<beast::ssl_stream<beast::tcp_stream>> m_sslUpStream;
beast::flat_buffer m_upBuffer;

void Session::do_read_upstream_buffered_response()
{
    beast::error_code ec;
    size_t bytes_transferred = 0
    beast::flat_buffer m_upBuffer;
    http::response_parser<http::buffer_body> resPar;
    resPar.body_limit(ULONG_MAX);

    bytes_transferred = http::read_header(*m_sslUpStream.get(), m_upBuffer, resPar, ec);

    if (ec)
    {
        return fail(ec, "read header");
    }

    http::response<http::buffer_body> bufRes;

    bufRes.result(resPar.get().result_int());
    bufRes.version(resPar.get().version());

    int field_count = 0;
    for (auto const& field : resPar.get())
    {
        bufRes.insert(field.name_string().to_string(), field.value().to_string());
    }

    // No data yet, but we set more = true to indicate
    // that it might be coming later. Otherwise the
    // serializer::is_done would return true right after
    // sending the header.
    bufRes.body().data = nullptr;
    bufRes.body().more = true;

    http::response_serializer<http::buffer_body, http::fields> resSer { bufRes };

    bytes_transferred = http::write_header(*(m_sslDownStream.get()), resSer, ec);

    if (ec)
    {
        LSPROXY_LOGD(LOG_MITM_PROXY, "Session[%d]::do_read_upstream_buffered_response(%s) Failed to write header to the original client (downstream) with error: %s", this, m_sessionStateStr, ec.message().c_str());
        return fail(ec, "write header");
    }

    // Read the rest of the response body upstream and send it downstream 
    while (!resPar.is_done())
    {
        char buf[8192];

        resPar.get().body().data = buf;
        resPar.get().body().size = sizeof(buf);

        bytes_transferred = http::read(*m_sslUpStream.get(), m_upBuffer, resPar, ec);

        if (ec == http::error::need_buffer)
        {
            ec.message().c_str());
            ec.assign(0, ec.category());
        }

        if (ec)
        {
            return fail(ec, "read body");
        }

        // Point to our buffer with the bytes that
        // we received, and indicate that there may
        // be some more data coming
        bufRes.body().data = buf;
        bufRes.body().size = sizeof(buf) - resPar.get().body().size;
        bufRes.body().more = true;

        bytes_transferred = http::write(*(m_sslDownStream.get()), resSer, ec);

        // This error is returned by body_buffer during
        // serialization when it is done sending the data
        // provided and needs another buffer.
        if (ec == http::error::need_buffer)
        {
            ec.message().c_str());
            ec = {};
            //continue;
        }

        if (ec)
        {
            return fail(ec, "write body");
        }

    } //while (!resPar.is_done())

    // `nullptr` indicates there is no buffer
    bufRes.body().data = nullptr;
    // `false` means no more data is coming
    bufRes.body().more = false;

    // Send the response header to the original client (downstream).
    bytes_transferred = http::write(*(m_sslDownStream.get()), resSer, ec);

    // Read another request from the original client (downstream)
    do_read_downstream();
}
marpas
  • 11
  • 1

1 Answers1

1

Just for the sake of others having the same or similar problem, I'd like to post the resolution to my problem. The answer was in the question all along. To be more precise in the https://www.boost.org/doc/libs/1_66_0/libs/beast/example/doc/http_examples.hpp there is an Example: HTTP Relay wich is exaclty what I needed in the first place. Theere are small differences in this example from what I have combined my self from the two other examples (mentioned in the original post). The most important one, the Example: HTTP Relay doesn't use a buffered body response:

http::response<http::buffer_body> bufRes;

to construct the serializer with:

http::response_serializer<http::buffer_body, http::fields> resSer { bufRes };

it uses the receiving parser directly to construct the serializer with:

// Create a parser with a buffer body to read from the input.
parser<isRequest, buffer_body> p;

// Create a serializer from the message contained in the parser.
serializer<isRequest, buffer_body, fields> sr{p.get()};

With a little adaptation, the Example: HTTP Relay works just fine for my proxy all types of requests, small body requsts, but also with big files downloading and data streams as well.

marpas
  • 11
  • 1