1

I have data incoming every minute which I need to write to several files. My solution at the moment is:

function writeData(data) {
   for(let i = 0; i < data.length; i++) {        
      stream = fs.createWriteStream('myFile' + [i] + '.txt', {flags: 'a'});
      stream.write(data[i] + '\n');
   }
}

writeData() is called every minute. Eventually my program crashes after some time. I don't remember the exact error, but it has to do with too many writeStreams/fds or similar.
Looking at my code, the error seems reasonable since I am creating every minute several new writeStreams. The question is, is there a way to create all streams only once and reference back to them (via ID or something) when needed?
Thanks in advance for your help.

unskilledDev
  • 43
  • 5
  • @RandyCasburn Because the number of streams depends on the size of `data`. I am looping through `data` to create a stream accordingly to `data` size. But unfortunately they are recreated every time i call `writeData()`. I don't know how to create them once and refer to them afterwards. – unskilledDev Mar 24 '21 at 14:59
  • Thanks - I didn't realize that for my first comment. You might consider forcing the close rather than relying on autoclose by calling `stream.end()` – Randy Casburn Mar 24 '21 at 15:02
  • @RandyCasburn If I had to close the streams every time, would it make any difference to just use `appendFile()` then? I learned that for frequent file handling, streams should be used to avoid excessive system open/close file commands. – unskilledDev Mar 24 '21 at 15:09
  • This: "_to avoid excessive system open/close file commands_" is predicated on the fact that you are continuously writing to the stream. in your case you are not - you are making a single write to the stream and ending. That's why I was confused in the beginning. Using the append flag makes no sense as you are only making a single write. In that regard then, you can use `.writeFile('myFile' + [i] + '.txt', data[I])`. – Randy Casburn Mar 24 '21 at 15:28
  • @RandyCasburn The files are created once and new data should be added to the existing content once every minute. So I think some function or some kind of a stream that appends the data, not overwrite it, is needed. I will try your approach to end the streams manually. – unskilledDev Mar 24 '21 at 15:35
  • OK...clearing my head here. I understand what you are trying to do and why that fails. – Randy Casburn Mar 24 '21 at 15:39
  • You create many, many, many write streams (one per second) for each file. That is the issue. You'll need a singleton (or module file) that: 1. checks the cache to see if the stream exists or creates each stream if not; 2. caches the stream reference for each file if not already cached, 3. Calls `writeData(data[I])` (that does not include the stream creation). – Randy Casburn Mar 24 '21 at 15:45

1 Answers1

1

You'll need to cache each stream and only create one if it doesn't exist in the cache. The easiest way to accomplish this is with a module (that acts as a singleton). The below code will get you on the right track.

// Module file for writeStream cacheing
const streams = [];

export default function writeData(data) {
  for (let i = 0; i < data.length; i++) {
    if (!inCache(i)) {
      addStream(fs.createWriteStream('myFile' + [i] + '.txt', {flags: 'a'}));
      addErrorHandler(i);
    }
    streams[i].write(data[i] + '\n');
  }
}

function inCache(idx) {
  return streams[idx] && streams[idx].writeable;
}

function addStream(stream) {
  streams.push(stream);
}

function addErrorHandler(idx) {
  stream[idx].on('error', (err) => console.log('do something'));
}
Randy Casburn
  • 13,840
  • 1
  • 16
  • 31