Avoiding memcpy in C++ buffered output

Question

I am trying to write directly to an fstream output buffer to avoid a memcpy.

Why does the following code not work?

It compiles, runs and produces the right length output file on Linux. But the output file does not contain correct text. Also note that for some reason, when I comment out the two lines involving str2, then an output file of zero length is produced.

Note: This example does not avoid memcpy, but if it works, it will help me avoid a memcpy in my application.

#include <fstream>

int main(int argc, char *argv[]) {
  std::fstream out;
  char buffer[512];
  out.rdbuf()->pubsetbuf(buffer, 512);
  out.open("file.txt", std::fstream::out);
  char *str1 = "test text.";
  strcpy(buffer, str1);
  out.rdbuf()->pubseekpos(strlen(str1), std::ios_base::out);
  char *str2 = "why?";
  out << str2;
  out.flush();
  out.close();
}

`to avoid a memcpy` -- sorry, but this is just silly. `memcpy` is one of the fastest operations. — Xeo, Apr 27 '11 at 04:36
There are several comments as to "why" I am doing it. Let's leave out those for the moment. Bottom line is that the code does not produce an output file with text: "test text. why?" Can someone tell me why? — Prasoon Tiwari, Apr 27 '11 at 05:25
@Xeo: Not only are you being annoying by answering a question that was not asked, you're also wrong. While memcpy() is fast, its cost is measurable and eliminating it can yield noticeable performance benefits, like a 10% benefit I measured in a parser I've written: http://article.gmane.org/gmane.comp.lang.lua.general/77955 — Josh Haberman, Apr 27 '11 at 05:59
@Josh: "by answering" - I put this as a comment exactly because it's not an answer. — Xeo, Apr 27 '11 at 06:03
If you want to access a file via a pointer I'd suggest just using a memory mapped file. — edA-qa mort-ora-y, Apr 27 '11 at 07:43
You're avoiding a `memcpy` but using a `strcpy` (and a `strlen`) ? What kind of sense does that make? — Chris Lutz, Apr 27 '11 at 21:14
Are you actually sure that the cost of the `memcpy` actually *matters* compared to the cost of the actual I/O being done? Have you profiled it? This may simply be a premature optimization and you should be concentrating your efforts elsewhere. — Mark B, Apr 27 '11 at 04:49

score 2 · Answer 1 · answered Apr 27 '11 at 06:29

You give the stream a buffer for its internal use. Then you don't give it anything to write.

The fact that you copy something to the buffer without telling the stream doesn't give you anything in the file.

As you might have noticed, the seekpos is for positioning in the file, not for moving in the buffer. The buffer is for the stream's internal use only!

score 1 · Answer 2 · answered Apr 27 '11 at 06:08

When you out.rdbuf()->pubseekpos(strlen(str1), std::ios_base::out);, this asks the stream to skip forwards through the newly created file. Can you expect it to take stuff from the buffer you specified? A clue: have you seen any examples involving specifying a buffer that said you needed to initialise it in some way or specify the initial number of non-garbage characters it contained? No... because the stream itself tracks which parts of the buffer are in use. Hence - when you take the presumed empty-buffer and skip forwards, it generates the intervening NULs. It wouldn't make sense for them to be set in the buffer (so simply doing a memcpy after the pubseekpos won't work either) - what if you've jumped more than a buffer-size forwards?

Hopefully this serves to at least illustrate the problem, although I haven't given any thought at this stage to how you might force the stream to change it's "tracking" of meaningful buffer content....

score 0 · Answer 3 · answered Apr 27 '11 at 04:39

0

It's still early, but aren't you basically doing

#include <fstream>

int main(int argc, char *argv[]) 
{

  std::fstream out;
  out.open("file.txt", std::fstream::out);

  out << "test text";
  out << "why?";

  out.flush();
  out.close();
}

?

answered Apr 27 '11 at 04:39

Mario The Spoon

4,799
1
24
36

In this example, yes! But can I get direct (read pointer) access to the output buffer? – Prasoon Tiwari Apr 27 '11 at 04:43
1

what do you want to achieve by that? since you do not call any write operation I am pretty sure that some implementation will not write anything. you are manipulating memory but not writing to a file. What is it you actually want to do? – Mario The Spoon Apr 27 '11 at 04:52
@user No, but I believe you can copy it into the heap. – Mateen Ulhaq Apr 27 '11 at 04:54

score 0 · Answer 4 · answered Apr 27 '11 at 06:04

I'm not familiar enough with the C++ iostream library to say what is wrong with your program, I would only mention that if you want to do your own buffering it would likely be more straightforward to use the read() and write() interfaces directly (man 2 write). If you're doing your own buffering the iostream libraries probably won't buy you much, and will only obscure what's actually going on.

score 0 · Answer 5 · answered Apr 27 '11 at 21:10

I see three possibilities

Implement your own stream-buffer class
Use the native file API
Disable buffering with pubsetbuf(0, 0)

Implementing a stream-buffer is really not that difficult, and would give you the opportunity to mix C++ style stream insertion with direct memory access, without sacrificing performance.

Avoiding memcpy in C++ buffered output

5 Answers5