NodeJS FS Write to Read inconsistent data without overflow (Solved as Buffer.toString)

Question

I'm using the fs.createWriteStream(path[, options]) function to create a write stream splitted in text lines each ending with \n.

But, when the process ended, if I go to check the stream leater it seems to be corrupted, showing some (few) corrupted lines (like a 0.05% of the lines looks partialy cutted like a buffer overflow error).

Anyway if I grow the internal stream buffer from 16k to 4M with the highWaterMark option at creation of the streem, the error rate seems to change but do not disappear!)

As the process goes on, errors are reported in kern.log, it could be due to a BMDMA stat 0x5 error? Is an hard drive DMA Buffer I/O transfer error, this could be the cause of errors in the stream? — Informate.it, Apr 01 '15 at 09:30

Informate.it · Answer 1 · 2015-04-01T12:16:55.343

It's due to an error in reading, not in writing. I was doing something like:

var lines=[],
 line='',
 buf=new Buffer(8192);

while ((fs.readSync(fd,buf,0,buf.length))!=0) {
 lines = buf.toString().split("\n");
 lines[0]=line+lines[0];
 line=lines.pop();

but this method you can find here and there on the web is really really wrong!

You have to check the real buffer lengt when you convert it to string, using buf.toString(null, 0 ,read_len)!!

var lines=[],
 line='',
 buf=new Buffer(8192),
 read_len;

while ((read_len=fs.readSync(fd,buf,0,buf.length))!=0) {
 lines = buf.toString(null, 0 ,read_len).split("\n");
 lines[0]=line+lines[0];
 line=lines.pop();

NodeJS FS Write to Read inconsistent data without overflow (Solved as Buffer.toString)

1 Answers1