fs.writeFile corrupting json files

Question

I have a program that writes a lot of files very quickly, and I noticed that sometimes there will be extra brackets or text in json files sometimes. Here is how the program works: There is an array of emojis with some more information, and if that emoji doesn't already have a file for itself, it creates a new one. If there already is an existing file of that name, it will edit it. Code to write file:

function writeToFile(fileName, file){
  return new Promise(function(resolve, reject) {
    fs.writeFile(fileName, JSON.stringify(file, null, 2), 'utf8', function(err) {
        if (err) reject(err);
        else resolve();
    });
  });
}

I have tried using fs and graceful-fs and both have had this issue every couple hundred files, with no visible patterns. examples off messed up json:

...
      ],
      "trade_times": []
    }
  ]
}ade_times": []
    }
  ]
}

That second "ade_times" shouldnt be there, and I have no idea why it is appearing. other times it just looks like this:

{
...
}}

with extra closing brackets for no reason. Not sure if this is an issue with my code, with fs, or something with my pc. If you need any more information I can provide that (more code, node.js version, etc). Thank you for your time :)

First off, you can just use `fs.promises.writeFile()` - you don't need to make your own promise function. Second, a common cause of corrupted files is multiple pieces of code or multiple requests trying to write to the same file at the same time. `fs.writeFile()` is not atomic. It consists of multiple separate asycnhronous operations `fs.open()`, `fs.write()` and `fs.close()` so other requests on your server can intervene and cause corrupted file. You may need to implement some sort of semaphore flag or promise that prevent this from happening. — jfriend00, Dec 31 '22 at 07:44

score -1 · Answer 1 · edited Dec 31 '22 at 10:42

I dont know why your code is buggy. But, I have created an alternate writeToFile function for you:

// Async functions are functions that return a promise in a more concise way.
async function writeToFile(filename, file) {
     let fj = JSON.stringify(file, undefined, 2)
     // fs.readFileSync will automatically reject the promise with the error when it encounters an error
     fs.writeFileSync(filename, fj, {
          encoding: "utf8"
     })
     return // This line is the equivalent of resolve(), this line is optional too.
}

Hope this answer works for you

fs.writeFile corrupting json files

1 Answers1