0

I'm currently writing a script that crawls a number of sites and dumps the results in a mongodb. When I run a bulk seed file (so I don't have to run each crawler and individual seed manually), The first crawler completes, but the following error hangs, preventing the rest of the steps from occuring.

fs: missing callback Error: ENOENT, open '../cache/nfHitters.json'

Below is my crawler code. Is there a way to remove this Error completely (or are there best practices I'm missing to prevent this from occuring all together)?

var cheerio = require("cheerio");
var request = require("request");
var json = require("json");
var fs = require('fs');

request('website-I-Am-Crawling.com', 
    function (error, response, html) {
        if (!error && response.statusCode == 200) {
            var $ = cheerio.load(html)
            var variable = $('script')[3].children[0].data
            var data = variable.substring(variable.indexOf("= ")+2, variable.indexOf(";"))
            fs.writeFile('../cache/nfHitters.json', data, error)
    };
});
mkayen
  • 27
  • 5

1 Answers1

2

error is not a function, but you're passing it as the callback argument to fs.writeFile(). If there is no error, error is null, so that is why you see that particular message. If it was some other non-null/undefined value, you would have gotten a type error thrown instead.

Try this:

fs.writeFile('../cache/nfHitters.json', data, function(err) {
  // ...
});
mscdex
  • 104,356
  • 15
  • 192
  • 153
  • when giving it a callback argument, the server hangs on the new callback (the function which console logs 'Theres an error'). So I guess my question is it defaulting to the callback, rather than completing the crawler action and moving on to the next script? – mkayen Sep 01 '15 at 04:30