1

I have some code that looks approximately like this:

boost::iostreams::filtering_istreambuf in;
in.push(Lz4DecompressionFilter());
in.push(AesDecryptionFilter());
in.push(file_source("somefile"));

I already have meta-data that stores the length of the result:

std::vector<char> buf;
buf.reserve(resultLength /* retrieved from a meta-data server */);
std::streamsize ret = in.read(buf, buf.capacity);

By adding trace-points, I observed that the Lz4 and Aes filter only get reads of 128 bytes. Also, if I replace file_source with a custom device, it only gets reads of 4096 bytes.

Since I know exactly the size the reads should have, is there a way to disable buffering in iostreams entirely and just chain the read down the filter? I know I can change the buffer sizes, but I am interested in completely disabling them.

user1202136
  • 11,171
  • 4
  • 41
  • 62

1 Answers1

0

  • Standard streams by definition use a buffer abstraction. This is largely because some of the functions exposed necessitate the presence of a buffer (peek/putback).

  • How would compression and encryption still function without buffering? Compression and block ciphers both require operating on (sometimes even fixed-size) chunks.

  • Re:

    Also, if I replace file_source with a custom device, it only gets reads of 4096 bytes.

    What behaviour would you have expected instead? Do you expect infinite size reads?

  • Using blocks of >4k is highly unusual in stream-oriented processing. In that case, did just just want to copy all input into one large buffer first (perhaps using array_sink...)?

Really, it looks like you just wanted to increase the buffer size yes.

sehe
  • 374,641
  • 47
  • 450
  • 633
  • I see your point. The thing is, both the AES filter and the LZ4 filter have quite complex buffering requirements, so they already have some buffers themselves. Hence, all my reads are already issued with the right buffer size (and location) and I do not want boost to do further, unnecessary memory copying. – user1202136 Mar 02 '15 at 17:29
  • Therefore the best solution would seem to be to modify the buffer size on the source to match your needs – sehe Mar 02 '15 at 17:33
  • I share your interest for efficiency, and in truth I might consider writing a filter that combines the steps here. In my case Boost Asio was involved so I employed zero-copy preallocated buffers. Admittedly that particular project is stuck in prototype mode :L – sehe Mar 02 '15 at 17:52