0

1st I read in a file line by line with my code. (Round about 1650 lines in the file)

2nd I reformate each line of the file into many lines.

3rd I would like to write the output in a new file. Unfortunately it doesn't write all of the more than 16800 lines. The output varied round about 15500 lines.

For 3rd I use folling code:

var inputArr; //Splited Input of one line 
var Text; //inputArr transformed to a String with many lines (per start line)
var lineCounter = 0; //counts the expacted number of output lines

const fs = require('fs');
const writeStream = fs.createWriteStream('./output.txt');

for(var i=0; i<= inputArr.length; i++) {
  writeStream.write(Text);

  lineCounter = lineCounter + 1;
}

writeStream.end();

What can I do to write all lines into my output file?

NewbieXXL
  • 155
  • 1
  • 1
  • 11

2 Answers2

0

What can I do to write all lines into my output file?

You can't write large amounts of data without detecting when the stream is full and then waiting for it to say its OK to write again. There's a pretty detailed example of how to do that in the stream.writable doc.

Here's an excerpt from the doc that shows how to do it:

// Write the data to the supplied writable stream one million times.
// Be attentive to back-pressure.
function writeOneMillionTimes(writer, data, encoding, callback) {
  let i = 1000000;
  write();
  function write() {
    let ok = true;
    do {
      i--;
      if (i === 0) {
        // last time!
        writer.write(data, encoding, callback);
      } else {
        // see if we should continue, or wait
        // don't pass the callback, because we're not done yet.
        ok = writer.write(data, encoding);
      }
    } while (i > 0 && ok);
    if (i > 0) {
      // had to stop early!
      // write some more once it drains
      writer.once('drain', write);
    }
  }
}

Basically, you have to pay attention to the return value from stream.write() and when it says the stream is full, you have to then restart writing on the drain event.

You don't show your whole code for both the reading and writing. If you're just reading a stream, modifying it and then writing the result to a different file, you should probably use piping, perhaps with a transform and then the streams will handle all the reading, writing and back pressure detection for you automaticallly.

You can read about transform streams here as this sounds like probably what you really want. You would then pipe the output of the transform stream to your output stream file and all the back pressure will be handled for you automatically.

jfriend00
  • 683,504
  • 96
  • 985
  • 979
  • Thanks. Pipe I figuered out, but seems not to work for me. I'm trying to understand the writeOneMillionTimes function (I'm not a developer). What is behind writer? Do I have to replace my for loop with that function? How can I adjust that function to my code? – NewbieXXL Nov 13 '18 at 08:40
  • @NewbieXXL - You can't expect to solve the problems you're trying to solve without learning how streams work and how to program them properly. `writer` in that function should be clear if you go look at the [actual documentation reference](https://nodejs.org/api/stream.html#stream_event_drain). It's a writable stream. Please read, study and learn. We don't just write finished code for you. We teach you where to find the information you need and you have to learn it yourself. As soon as `.write()` returns `false`, you can not write any more until the `drain` event is fired. – jfriend00 Nov 13 '18 at 08:45
  • @NewbieXXL - Your reference to "I'm not a developer" just means you're going to have to learn to be a developer to write code like this. Depending upon the context for this code, there are simpler ways to write data to a file such as `fs.writeFile()`, `fs.writeSync()`, etc... that might or might not be appropriate too. – jfriend00 Nov 13 '18 at 08:47
  • I like to learn! I would like to understand the nodejs documentation. But it isn't self-explanatory for me as newbie. I don't recognize the import details for successful streams. I have written lots of code and delete, because it didn't work. I have experimented with buffer, your example and fs.writeFile() beforehand. Without succes, that's why I'm posting here. – NewbieXXL Nov 13 '18 at 08:57
  • @NewbieXXL - I'd suggest you back up and explain (in your question using the "edit" button) what your top level problem is that you're trying to solve. Then, and only then, could we perhaps suggest a better or simpler way to approach the problem. You can't write lots of data to a stream without paying attention to when the stream is full. You just can't. – jfriend00 Nov 13 '18 at 09:01
  • '@jfriend00 - I understood your point ". You can't write lots of data to a stream without paying attention to when the stream is full." Now I try to realize that in my code. My top level problem I have tried to discribe above. How does it merge with your recommendation? – NewbieXXL Nov 13 '18 at 09:08
  • @NewbieXXL - There's not enough detail in your question about the top level problem. We need to see what you're reading, what you're doing when you convert it to something else and understand how big the overall data set is (whether in-memory solutions are feasible or not). We also need to know the environment you're running this in to know whether synchronous I/O is an option (is this just a script to do data conversion or part of a multi-user server or what?). Basically, you have to give us more context and help us understand so we can better help you. – jfriend00 Nov 13 '18 at 09:11
  • I have found my problem. Synchron code stumpled on asynchron code or vice versa. Thanks for the hint, that my problem could be in another part of my code, jfriend00. (How can you write @ nick here? If I try, this side deletes it.) – NewbieXXL Nov 15 '18 at 10:55
0

how I use the write

const { finished } = require('node:stream')
const { once } = require('events')
const fs = require('fs')

async function writeWithAwait(writable, chunk) {
  if (!writable.write(chunk)) {
    // Handle backpressure
    await once(writable, 'drain')
  }
}

function finishedAwait(stream) {
  return new Promise((resolve, reject) => {
    finished(stream, err => {
      if (err) {
        reject(err)
      } else {
        resolve()
      }
    })
  })
}

const writer = fs.createWriteStream(`/tmp/my.xml`)

writer.on('error', err => console.error(err))

writer.on('open', async function() {
  await writeWithAwait(writer, `text`)
  // writer.on("end", () => {
  //   ...why not called?
  // })
  writer.end() // TODO: do I need to call it?
  const maybeFinishError = await finishedAwait(writer)
  if (maybeFinishError) {
    console.error('Stream failed.', maybeFinishError)
    return
  }
  console.log('Stream is finished.');
  // do other stuff
})
srghma
  • 4,770
  • 2
  • 38
  • 54