Writable Stream empties the file but do not write expected content in nodeJS

Question

This is my code to get an array of records and write them as CSV string one by one in a file. I cannot get the writable stream to write in the file. It just erases whatever is in the file and does not write the new data in.

const writeStream = fs.createWriteStream(process.env.SAVED_FILE);
const transformStream = new Stream.Transform({
    writableObjectMode: true,
    readableObjectMode: true,
})
transformStream._transform = (chunk, encoding, callback) => {
    console.log(chunk);
    transformStream.push("\"" + chunk.join("\",\"") + "\"");
    callback();
}
const readable = new Stream.Readable({objectMode: true})
    .pipe(transformStream)
    .pipe(writeStream);
let records = [["a", "b"], [1, 2], ["c", "d"], [8, 9]];
records.forEach(record => readable.push(record));
readable.push(null);

writeStream.on("error", function (err) {
    console.log("there is an error!")
});
writeStream.on("end", function () {
    console.log("reached the end of the stream!")
});

Any help appreciated on this basic example !

`createWriteStream` creates a stream that by default writes *at the beginning of the file*, effectively replacing it: http://nodejs.org/dist/latest-v14.x/docs/api/fs.html#fs_fs_createwritestream_path_options explains the `start` and `flags` options, specifically that by default the file is replaced (`flags` is `w` by default) and `start` is (possibly) implied to be zero by default. — Armen Michaeli, Jun 05 '22 at 13:05
res is the result that node will send back to the user - standard node but not useful here indeed — Chapo, Jun 06 '22 at 01:23

score 2 · Accepted Answer · answered Jun 05 '22 at 13:12

2

pipe() method returns the destination stream, i.e. the writable stream.

In your code,

const readable = new Stream.Readable({objectMode: true})
    .pipe(transformStream)
    .pipe(writeStream);

readable doesn't contains a reference to a readable stream like you expect. Instead, it refers to the last stream in the chain, i.e. writeStream and you cannot call push on a writable stream.

Change your code as shown below:

const readable = new stream.Readable({ objectMode: true });

readable
  .pipe(transformStream)
  .pipe(writeStream);

answered Jun 05 '22 at 13:12

Yousaf

27,861
6
44
69

Thank you very much it works ! another question if you have 2 minutes: my "end" event on the `writeStream` never gets triggered. How should I handle that ? I thought pushing "null" to a readable stream triggered the "end" event immediately ? – Chapo Jun 06 '22 at 01:31
1

Writable streams don't have an event named `end` (readable streams do). Writable streams have a `close` event. – Yousaf Jun 06 '22 at 05:26
1

Thank you very much. Now i'll go read the fricking docs... – Chapo Jun 06 '22 at 08:05

Writable Stream empties the file but do not write expected content in nodeJS

1 Answers1