Node.js how to wait on asynchronous call (readdir and stat)

Question

I am working on post method in the server side to retrieve all files inside the requested directory (not recursive), and below is my code.

I am having difficulty sending the response back (res.json(pathContent);) with the updated pathContent without using the setTimeout.

I understand that this is due to the asynchronous behavior of the file system methods used (readdir and stat) and need to use some sort of callback, async, or promise technique.

I tried to use the async.waterfall with the entire body of readdir as one function and the res.json(pathContent) as the other, but it didn't send the updated array to the client side.

I know that there have been thousands of questions regarding this asynchronous operation but could not figure out how to solve my case after reading number of posts.

Any comments would be appreciated. Thanks.

const express = require('express');
const bodyParser = require('body-parser');
const fs = require('fs');
const path = require('path');

var pathName = '';
const pathContent = [];

app.post('/api/files', (req, res) => {
    const newPath = req.body.path;
    fs.readdir(newPath, (err, files) => {
        if (err) {
            res.status(422).json({ message: `${err}` });
            return;
        }
        // set the pathName and empty pathContent
        pathName = newPath;
        pathContent.length = 0;

        // iterate each file
        const absPath = path.resolve(pathName);
        files.forEach(file => {
            // get file info and store in pathContent
            fs.stat(absPath + '/' + file, (err, stats) => {
                if (err) {
                    console.log(`${err}`);
                    return;
                }
                if (stats.isFile()) {
                    pathContent.push({
                        path: pathName,
                        name: file.substring(0, file.lastIndexOf('.')),
                        type: file.substring(file.lastIndexOf('.') + 1).concat(' File'),
                    })
                } else if (stats.isDirectory()) {
                    pathContent.push({
                        path: pathName,
                        name: file,
                        type: 'Directory',
                    });
                }
            });
        });
    });    
    setTimeout(() => { res.json(pathContent); }, 100);
});

Answer 1

The easiest and cleanest way would be use await/async, that way you can make use of promises and the code will almost look like synchronous code.

You therefor need a promisified version of readdir and stat that can be create by the promisify of the utils core lib.

const { promisify } = require('util')

const readdir = promisify(require('fs').readdir)
const stat = promisify(require('fs').stat)

async function getPathContent(newPath) {
  // move pathContent otherwise can have conflicts with concurrent requests
  const pathContent = [];

  let files = await readdir(newPath)

  let pathName = newPath;
  // pathContent.length = 0;  // not needed anymore because pathContent is new for each request

  const absPath = path.resolve(pathName);

  // iterate each file

  // replace forEach with (for ... of) because this makes it easier 
  // to work with "async" 
  // otherwise you would need to use files.map and Promise.all
  for (let file of files) {
    // get file info and store in pathContent
    try {
      let stats = await stat(absPath + '/' + file)
      if (stats.isFile()) {
        pathContent.push({
          path: pathName,
          name: file.substring(0, file.lastIndexOf('.')),
          type: file.substring(file.lastIndexOf('.') + 1).concat(' File'),
        })
      } else if (stats.isDirectory()) {
        pathContent.push({
          path: pathName,
          name: file,
          type: 'Directory',
        });
      }
    } catch (err) {
      console.log(`${err}`);
    }
  }

  return pathContent;
}

app.post('/api/files', (req, res, next) => {
  const newPath = req.body.path;
  getPathContent(newPath).then((pathContent) => {
    res.json(pathContent);
  }, (err) => {
    res.status(422).json({
      message: `${err}`
    });
  })
})

And you should not concatenated paths using + (absPath + '/' + file), use path.join(absPath, file) or path.resolve(absPath, file) instead.

And you never should write your code in a way that the code executed for the request, relays on global variables like var pathName = ''; and const pathContent = [];. This might work in your testing environment, but will for sure lead to problems in production. Where two request work on the variable at the "same time"

Answer 2

Based on the initial comment I received and the reference, I used readdirSync and statSync instead and was able to make it work. I will review other answers as well and learn about other ways to implement this.

Thank you all for your kind inputs.

Here is my solution.

const express = require('express');
const bodyParser = require('body-parser');
const fs = require('fs');
const path = require('path');

var pathName = '';
const pathContent = [];

app.post('/api/files', (req, res) => {
    const newPath = req.body.path;

    // validate path
    let files;
    try {
        files = fs.readdirSync(newPath);
    } catch (err) {
        res.status(422).json({ message: `${err}` });
        return;
    }

    // set the pathName and empty pathContent
    pathName = newPath;
    pathContent.length = 0;

    // iterate each file
    let absPath = path.resolve(pathName);
    files.forEach(file => {
        // get file info and store in pathContent
        let fileStat = fs.statSync(absPath + '/' + file);
        if (fileStat.isFile()) {
            pathContent.push({
                path: pathName,
                name: file.substring(0, file.lastIndexOf('.')),
                type: file.substring(file.lastIndexOf('.') + 1).concat(' File'),
            })
        } else if (fileStat.isDirectory()) {
            pathContent.push({
                path: pathName,
                name: file,
                type: 'Directory',
            });
        }
    });
    res.json(pathContent);
});

Answer 3

Here's some options:

• Use the synchronous file methods (check the docs, but they usually end with Sync). Slower, but a fairly simple code change, and very easy to understand.
• Use promises (or util.promisify) to create a promise for each stat, and Promise.all to wait for all the stats to complete. After that, you can use async functions and await as well for easier to read code and simpler error handling. (Probably the largest code change, but it will make the async code easier to follow)
• Keep a counter of the number of stats you have done, and if that number is the size you expect, then call res.json form inside the stat callback (smallest code change, but very error prone)

Answer 4

There is different way to do it :

1. You can first promisify the function with using new Promise() then second, use async/await or .then()
2. You can use the function ProsifyAll() of the Bluebird package (https://www.npmjs.com/package/bluebird)
3. You can use the synchrone version of the fs functions
   • https://nodejs.org/api/fs.html#fs_fs_readdirsync_path_options
   • https://nodejs.org/api/fs.html#fs_fs_statsync_path_options