asio::async_read_until: robust and graceful way of handling multiple lines

Question

I'm using asio::async_read_until with '\n' delimiter to support a TCP client that fetches character data from a server. This server continuously sends '\n' terminated lines; precisely, it can write at once either single lines or a concatenated string of multiple lines.

From the doc, I understand that asio::async_read_until could read:

• One '\n' terminated line, like "some_data\n". This is the simplest case, handled with a call the std::getline on the stream associated with the asio::streambuf
• One '\n' terminated line plus the beginning of a next line, like "some_data1\nbla". This can be handled with a std::getline; the rest of the second line will be handled at the next completion handler call.
• Many lines; in this case, the newly read data could contain 2 or more '\n'. How can I know how many std::getline calls I should do, knowing that I don't want to risk calling std::getline on an incomplete line (which I will eventually get in a future packet)? Should I peek at the stream buffer to check the existence of multiple '\n'? Is it even possible without doing many copies?

Answer 1

from the documentation here:

http://www.boost.org/doc/libs/1_59_0/doc/html/boost_asio/reference/async_read_until/overload1.html

If the stream buffer already contains a newline, the handler will be invoked without an async_read_some operation being executed on the stream.

For this reason, when your handler executes you must execute no more than one getline(). Once getline has returned and you have finished processing, simply call async_read_until again from the handler.

example:

void handler(const boost::system::error_code& e, std::size_t size)
{
  if (e)
  {
    // handle error here
  }
  else
  {
    std::istream is(&b);
    std::string line; 
    std::getline(is, line);
    do_something(line)
    boost::asio::async_read_until(s, b, '\n', handler);
  }
}

// Call the async read operation
boost::asio::async_read_until(s, b, '\n', handler);

Answer 2

this answer relates to the accepted answer:

I'd highly recommand to call std::getline() in a loop and test the return value.

while (std::getline(is, line)) {
  ...
  do_something(line);
}

std::getline returns a reference to the istream reference, which can be implicitely converted to bool, indicating if the getline operation was really successful.

Why one shall do that:

1. std::getline may fail, i.e. if the input stream has reached its limits, and no newline is present
2. you may have more then one line inside asio's streambuf. If you blindly restart reading after processing just the first line, you may end up with exceeding memory limits on the streambuf (or have an ever growing streambuf).

Update 2017-08-23:

bytes_transferred actually gives you the position in the underlying buffer where the separator has been found. One can take advantage of that by simply upcasting the streambuf and create a string from that.

void client::on_read(const std::error_code &ec, size_t bytes_transferred) {

    if (ec) {
        return handle_error(ec);
    }

    std::string line(
        asio::buffer_cast<const char*>(m_rxbuf.data()),
        bytes_transferred
    );

    // todo: strip of trailing delimiter

    m_rxbuf.consume(bytes_transferred); // don't forget to drain

    handle_command(line); // leave restarting async_read_until to this handler
}

instead of copying data from the streambuf into the string, you can alternatively create a string_view from it, or replace the underlying streambuf by a std::string and chop off the bytes_transferred instead of consuming from the buffer.

Cheers, Argonaut6x

Answer 3

Updated: with a somewhat better approach.

IMHO, you are better off using async_read_some directly rather than the read until operation. This requires less operations overall and gives you better control over the buffer handling, and could reduce the amount of copies you have to make of the data. You could use the asio::streambuf implementation, but you could also do this using a vector<char>, for example:

vector<char> buffer(2048); // whatever size you want, note: you'll need to somehow grow this if message length is greater...
size_t content = 0; // current content

// now the read operation;
void read() {
  // This will cause asio to append from the last location
  socket.async_read_some(boost::asio::buffer(buffer.data() + content, buffer.size() - content), [&](.. ec, size_t sz) {
    if (ec) return; // some error
    // Total content in the vector
    content += sz;
    auto is = begin(buffer);
    auto ie = next(is, content); // end of the data region

    // handle all the complete lines.
    for (auto it = find(is, ie, '\n'); it != ie; it = find(is, ie, '\n')) {
      // is -> it contains the message (excluding '\n')
      handle(is, it);
      // Skip the '\n'
      it = next(it);
      // Update the start of the next message
      is = it;
    }
    // Update the remaining content
    content -= distance(begin(buffer), is);
    // Move the remaining data to the begining of the buffer
    copy(is, ie, begin(buffer));
    // Setup the next read
    read();
  });
}