createWriteStream instance shadowing another instance in nodeJS

Question

My aim is to do something like that :

function writer1(data,file){
    const w = fs.createWriteStream(file,{flags:'w'})
    for(let i = 0; i< data.length; i++){
       w.write(data[i])
    }
    w.end()
}

function writer2(data,file, *some-stuff*){
    const w = fs.createWriteStream(file,{flags:'w'})
    for(let i = 0; i< data.length; i++){
        if(data[i] !== *some-stuff*){
              w.write(data[i])
        }
    }
    w.end()
}

writer1(data,"file.txt")
writer2(data,"file.txt", "some string")

IMPORTANT TO NOTE : in the true piece of code I'm writing, writer1 has a condition to run; it runs only if the file it needs to write does not exists

But here is my problem; if the according files does not exists, i.e. if the 'STATE' of the project is init-state, then writer1 is launched but somehow shadows the execution of writer2. The result is a txt file filled with the content DATA.

On the second pass, then writer1 is not launched, does not shadow the execution of writer2, and the result is a txt file filled with the content of DATA MINUS the variable some-stuff.

Essentially, my question is :

Why is the first stream shadowing the second and how to prevent that ?

I do understand that there's something asynchronous to be dealed with or a request to be made to the stream object in order to allow for other streams to access the same file. What is missing ?

`w.write()` returns a boolean that indicate when the buffer is full and when you have to wait for the `drain` event before doing further writing. If you're writing a lot, you will need to pay attention to this aspect of writing to a writeStream. — jfriend00, Jan 20 '23 at 17:46
So, you simply can't do uncoordinated writing to two writeStreams that point at the same file. That's just a recipe for disaster. I don't know what the real problem is you're trying to solve here (unsure what "shadowing" is), but what you show will not accomplish anything useful. I would suggest you back up to the beginning and describe what the original problem is you're trying to solve here because this is not the way to do it. — jfriend00, Jan 20 '23 at 17:47
The real problem I'm trying to solve is generating and updating files on demand. Naively I though writestreams could be used as disposable streams with precise lifetimes and that I could "know" when one of them closes in order to control the flow of my program. — AVALFINN, Jan 20 '23 at 17:55
writeSteams work just fine for writing files. They do require following a few rules such as paying attention to the flow control return value from `.write()` and coding that properly. Still not sure what you're actually asking for help here with? There are lots of examples of how to properly code `.write()` in the documentation. Also, streams are asynchronous so coding with them has to be aware of that too. For example `.end()` doesn't immediately finish. It has a callback that tells you when it's actually done. — jfriend00, Jan 20 '23 at 17:58
We can't see your real code (not sure why you didn't include the REAL code), but I would guess you are trying to start the second write BEFORE the first one is complete and thus the two collide and are both writing to the file at the same time and one overwrites the other creating a mess in the file. — jfriend00, Jan 20 '23 at 18:02
Stream one prevents stream two to write the same file : how do I wait for stream one to end before running stream two ? — AVALFINN, Jan 20 '23 at 18:04
You either use the callback you can pass to `w.end()` or you use the `close` event on the stream. Only when you get those has the first stream finished. And, as I've said a couple times, you should pay attention to the return value from `.write()`. Streams are ASYNCHRONOUS and you must be aware of that in how you code with them. — jfriend00, Jan 20 '23 at 18:08

Heiko Theißen · Answer 1 · 2023-01-20T17:48:32.613

Writing to a stream is an asynchronous process. If you open the file again in writer2 before writer1 has closed it, the writings of writer2 may be lost.

The following variant of writer1 is an asynchronous function that resolves only after closing the file. You can await this before calling writer2.

function writer1(data, file) {
  return new Promise(function(resolve, reject) {
    const w = fs.createWriteStream(file, {flags: 'w'})
    .on("close", resolve)
    .on("error", reject);
    for (let i = 0; i < data.length; i++)
      w.write(data[i]);
    w.end();
  });
}
function writer2(...) // similar
await writer1(data, "file.txt");
await writer2(data, "file.txt", "some string");

But I second jfriend00's question about what problem you are trying to solve.

createWriteStream instance shadowing another instance in nodeJS

Why is the first stream shadowing the second and how to prevent that ?

1 Answers1