0

Suppose I have a buffer with length 100MB char[100*1024*1024]. I would like to use stringstream facilities on this buffer like format read so I defined a new stringstream ss(arr) using this array. So I would like to learn if my program uses 200MB in total at runtime or not? I am working on big data and memory usage is critical. Actually I defined a char buffer and initialize my custom istream with this buffer and solved my memory issue. But I am still in confusion whether my second way is redundant or not.

mustafa.yavuz
  • 1,274
  • 2
  • 21
  • 40
  • Why don't you simply *measure* it yourself? – Jesper Juhl Mar 31 '17 at 16:29
  • I tried by changing original buffer and as it is expected it did not effect stringstream. Using other tools like valgrind, xcode I could not get it they show no info about internal mechanism of stringstream. Since it is a critical issue I just would like to be sure when stringstream is initialized with and array is there a memory allocation as size of buffer. – mustafa.yavuz Mar 31 '17 at 16:39
  • 2
    Pretty sure you're going to have duplication. I don't read any any certainty from the standard, but `stringstream` contains a `stringbuf`, and `stringbuf` usually requires a resizable character sequence. Your array isn't resizable. What will probably happen is your 100 MB buffer will be converted to a `std::string` and the `string` will form the basis of the `stringbuf`. – user4581301 Mar 31 '17 at 16:55
  • Have a look at boost::iostream's [array device](http://www.boost.org/doc/libs/release/libs/iostreams/doc/classes/array.html). It allows you to treat an array as a stream without doing any new allocations. The array device doesn't own the array in contrast to `std::stringstream`. – zett42 Mar 31 '17 at 19:46

1 Answers1

1

So I would like to learn if my program uses 200MB in total at runtime or not?

If you construct a stringstream from a char array it will at least double the memory usage. From the reference of std::basic_stringstream constructor:

Uses a copy of str as initial contents of the underlying string device.

You could write your own stream buffer to create a non-owning stringstream, but the boost::iostreams library already provides that via array devices. 

namespace io = boost::iostreams;

char str[100*1024*1024];

// No copy of str is made here! The stream just stores pointer and size of array.
io::stream<io::array_source> strm( str, sizeof(str) );

// Do something with the stream as usual. It is fully compatible with standard streams.
int x;
strm >> x;

There is a nice online compiler that shows peak memory usage. In the following examples I'm creating a stream from a 10 MiB array.

Live example using std::stringstream. Peaks at 31(!) MiB.

Live example using boost::array_source. Peaks at only 11 MiB.

Why there is a peak memory usage of even 3x the char array size when using std::stringstream? Because we must create a temporary string from the char array first as std::stringstream doesn't have a constructor that takes a char pointer.

zett42
  • 25,437
  • 3
  • 35
  • 72