Read multiple files from directory with Node.js using Promises/Async/Await

Question

I'm trying to scan a local directory for all of the text files within it, then read each of those files. I want to do it properly using promises, async, and await, but something is going wrong and am stuck trying to figure it out.

I am able to read the directory contents and output all of the filenames correctly, but when I try to read those files themselves using the map method, I get a Promise { <pending> } log for each of the files. Also, I get a UnhandledPromiseRejection error at the end.

I can read a single file fine outside of the map function, but can't loop through all of the files and read them without getting an error.

import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const textFileDir = `${__dirname}/files/`;

const readDirPro = (dir) => {
  return new Promise((resolve, reject) => {
    fs.readdir(dir, (error, files) => {
      if (error) reject('Could not find directory');
      resolve(files);
    });
  });
};

const readFilePro = (file) => {
  return new Promise((resolve, reject) => {
    fs.readFile(file, 'utf-8', (err, data) => {
      if (err) reject('Could not find file');
      resolve(data);
    });
  });
};

const readAllFiles = async () => {
  try {
    const dirFilesArr = await readDirPro(textFileDir);
    // correctly outputs array of file names in directory
    // console.log('dirFilesArr: ', dirFilesArr);

    // THIS IS THE PROBLEM:
    await dirFilesArr.map((file) => {
      // console.log(file);
      // correctly outputs filenames
      const fileContent = readFilePro(file);
      // console.log(fileContent);
      // incorrectly outputs "Promise { <pending> }" logs for each file
    });
  } catch (err) {
    console.log(`catch (err): ${err}`);

    throw err;
  }
  return '2: final return';
};

(async () => {
  try {
    console.log('1: readAllFiles!');
    const x = await readAllFiles();
    console.log(x);
    console.log('3: Done!');
  } catch (err) {
    console.log(`IIFE catch ERROR : ${err}`);
  }
})();

Here's is the full console output for that javascript, with the `Promise { } logs commented out:

1: readAllFiles!
2: final return
3: Done!
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "Could not find file".] {
  code: 'ERR_UNHANDLED_REJECTION'
}

Node.js v19.0.1

I would prefer to keep the working solution as close to this code as possible, because it's is actually one piece of a larger puzzle, and I need to be able to continue utilizing this structure.

My ultimate goal is to read all of those files, pull certain data from them, then write that data into a new JSON file.

UPDATE: Check this answer below for the most exact solution to the question: https://stackoverflow.com/a/75206022/3787666

`readFilePro()` returns a pending promise. That's what it does. The caller has to use `await` or `.then()` on that promise to know when it's done and get the result. — jfriend00, Jan 22 '23 at 06:39
Do you realize that promise versions already exist for the `fs` module? See [fsPromises.readFile()](https://nodejs.org/api/fs.html#fspromisesreadfilepath-options) in the doc. — jfriend00, Jan 22 '23 at 06:51
thanks, @jfriend00, but where does that await go? I have one before the map method here, and if I try it like this `const fileContent = await readFilePro(file);`, then I get an "Unexpected reserved word" error — david, Jan 22 '23 at 07:01
As for fsPromises.readFile(), are you saying I could use that instead of the readFilePro function here? — david, Jan 22 '23 at 07:04
Yes, it's built into the `fs` module. You don't have to make your own promisified versions. — jfriend00, Jan 22 '23 at 07:11
I'm still not clear on how that helps with my issue here. I need to be able to loop through all of the files and read them. I have to be able to read the directory then dynamically read each file in that directory. — david, Jan 22 '23 at 07:39
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/251315/discussion-between-david-and-jfriend00). — david, Jan 22 '23 at 08:11
Maybe this link can help you : https://medium.com/@leonardobrunolima/javascript-tips-asynchronous-iteration-7a13649a3348 — phili_b, Jan 22 '23 at 08:26
Thanks, @phili_b. Definitely in the right ball park, figuring out how to await promises with a map method. — david, Jan 22 '23 at 10:02

score 5 · Accepted Answer · answered Jan 22 '23 at 08:35

NodeJS has promisified version of fs, so technically you can achieve the desired results with the following code:

import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const textFileDir = `${__dirname}/files/`;

(async () => {
  try {
    console.log('1: readAllFiles!');
    const dirFilesArr = await fs.promises.readdir(textFileDir);
    const files = await Promise.all(dirFilesArr.map((file) => {
        const fullPath = path.join(textFileDir, file);
        return fs.promises.readFile(fullPath, { encoding: 'utf-8' });
    }));
    console.log(files);
    console.log('3: Done!');
  } catch (err) {
    console.log(`IIFE catch ERROR : ${err}`);
  }
})();

If you prefer to fix your original code, you will need to make a few changes. First change is needed because readdir doesn't return file names with full pathes and as a result, you need to add a path to each file to read a file. Therefore, your readFilePro should look something like so:

const readFilePro = (file) => {
  return new Promise((resolve, reject) => {
    fs.readFile(path.join(textFileDir, file), 'utf-8', (err, data) => {
      if (err) reject('Could not find file');
      resolve(data);
    });
  });
};

The second change is that your readAllFiles needs to be changed like so:

const readAllFiles = async () => {
  try {
    const dirFilesArr = await readDirPro(textFileDir);
    return Promise.all(dirFilesArr.map((file) => readFilePro(file)));
  } catch (err) {
    console.log(`catch (err): ${err}`);

    throw err;
  }
  return '2: final return';
};

In your original code, you run readFilePro for each file and retrieve a promise that you never wait. The approach above fixes the problem.

Please let me know if this helps.

Thank you so much! Providing two solutions even. I was just looking at handling the `map` method with a `Promise.all` and that turns out to be a key part of the solution. — david, Jan 22 '23 at 09:59

david · Answer 2 · 2023-01-24T02:57:22.477

Building on the accepted answer above, here's a supplemental answer, which addresses a key aspect of my question. Rather than returning all of the data in in one big chunk from the map, I really needed to be able to manipulate data from each file as soon as it is read (why I had the const fileContent in my question above). So, this is how I handle that now, by using an async/await for the callback inside of the map method. I still need the Promise.all method with the return if I want to pass const files data to a function following the Promise.all.

const files = await Promise.all(
      dirFilesArr.map(async (file, i) => {
        const fullPath = path.join(textFileDir, file);
        const fileContent = await fs.promises.readFile(fullPath, {
          encoding: 'utf-8',
        });
        //console.log(fileContent); // can now view file data individually
        await externalFunction(fileContent, i); // manipulating file data
        // return fs.promises.readFile(fullPath, { encoding: 'utf-8' });
      })
    );

@RAllen Your super helpful answer got me to this point. Thanks again — david, Jan 23 '23 at 06:14

Read multiple files from directory with Node.js using Promises/Async/Await

2 Answers2