Asynchronously writing files in a loop, how to manage streams

Question

I am trying to write over 100 png files via node-canvas in a loop. Only 40 files get generated and then the process completes.

I have tried creating a png stream via createPNGStream() and piping the results to a write stream created by fs.createWriteStream().

Write function:

function writeFile(row, col) {
        // canvas code ...
        const stream = canvas.createPNGStream();
        let path = __dirname + '/imgs/heatmapRow' + row + "Col" + col + '.png';
        const out = fs.createWriteStream(path);
        stream.pipe(out);
        out.on('finish', () => console.log('The PNG file was created.'))
    }

Calling function:

function generateImages(){
    var numRows = 20;
    var numCols = 5;
    for(let row = 0; row < numRows; ++row) {
        for (let col = 0; col < numCols; ++col) {
            writeFile(row, col);
        }
    }
}

The loop runs and completes, and at the end I get a bunch of the following lines all at once:

The PNG file was created. The PNG file was created. The PNG file was created. The PNG file was created.

I'm thinking that on each loop a writeStream is being created and is asynchronous. The process is terminating because I can have only so many streams open.

How can I write all of my files and do so asynchronously to speed up the process time? (I prefer not to write the files synchronously) Do I need to add each writeFile to a queue? How do I determine limit the number of streams I have open and how do I manage them?

So basically only 40/100 files are generated?. Do you see any error?. — Marcos Casagrande, Mar 28 '19 at 13:41
It’s about 40 images. One time it was 37, one time it was 43. I didn’t have any errors. — afriedman111, Mar 28 '19 at 13:53

score 3 · Answer 1 · edited Mar 28 '19 at 17:26

You have to use Promises to make your asynchronous calls. I give you a solution:

  function writeFile(row, col) {
    // canvas code ...
    const stream = canvas.createPNGStream();
    let path = __dirname + "/imgs/heatmapRow" + row + "Col" + col + ".png";

    return new Promise(resolve => {
      const out = fs.createWriteStream(path);
      stream.pipe(out);
      out.on("finish", () => {
        console.log("The PNG file was created.");
        resolve();
      });
    });
  }

  function generateImages() {
    var numRows = 20;
    var numCols = 5;
    var imagesToGenerate = [];
    for (let row = 0; row < numRows; ++row) {
      for (let col = 0; col < numCols; ++col) {
        imagesToGenerate.push(writeFile(row, col));
      }
    }

    Promise.all(imagesToGenerate).then(() => {
      console.log("All images generated");
    });
  }

Take a look at Promise.all docs if you don't clearly understand how it is working

Asynchronously writing files in a loop, how to manage streams

1 Answers1