0

I am trying to write a file copy code in Node.js: 

function archiveFolder(site, callback) {
//console.log("Processing Site2 " + site );
var path = srcDir + '/' + site;
var startDate = moment(new Date(), 'YYYYMMDD');
asyncCounter++;
console.log("getting list of files for site " + site);
fse.readdir(path, function(err, files) {
    console.log("got list of files for site " + site);
    if (err) {
        console.log(err);
        return callback(err);
    }
    async.eachLimit(files, 5, function(file, callback2) {

        var endDate = moment(file, 'YYYYMMDD');
        var diff = startDate.diff(endDate, 'days');
        fs.stat(path + '/' + file, function(err, stats) {
            try {
                if (err) {
                    return callback2(err);
                }
                if (/^\d{8}$/i.test(file) && stats.isDirectory() && diff >= 7) {
                    fse.ensureDir(destDir + '/' + site, function(err) {
                        fse.copy(path + '/' + file, destDir + '/' + site + '/' + file, function(err) {
                            try {
                                if (err) {
                                    //console.log(err);
                                    return callback2(err);
                                } else {
                                    console.log("copied " + site + '/' + file);
                                    return callback2();
                                }
                            } catch (e) {
                                console.log(e);
                                return callback2(err);
                            }
                        });
                    });

                } else {
                    // //console.log("failed " + path + '/' + file)
                    return callback2();
                }
            } catch (e) {

                console.log(e);
                return callback2(err);
            }

        });


    }, function(err) {
        if (!err) {
            //console.timeEnd("site");
            console.log("finished Site " + site + " asyncCounter " + asyncCounter);
            return callback();

        } else {
            return callback(err + "  test");
        }
    });
});

}

Still there isn't any error message on console or the system logs /var/log/messages

What could be the reason for this behavior? I can increase the ulimit on the server but I want to confirm that this is due to this "number of files open" limit.

Sravan
  • 596
  • 1
  • 9
  • 19
  • Since this is asynchronous, if you're trying to startup 70k `fse.copy()` commands at once, you are probably exhausting some system resource by trying to initiate that many file operations at once. Since it does absolutely no good to run more than a few copies at a time, you should probably run only run a couple at a time and each time one finishes, you start the next one. – jfriend00 May 12 '17 at 19:38
  • If you can show us to calling code that calls `fse.copy()`, then we can probably advise better on how to fix your code to only run a few of these at a time. You don't want to try to fix this with `ulimit` since there's no reason to ever even come close to that limit. You're just trying to run too many simultaneous operations at once when there is no benefit in doing so. It just unduly stresses the system and probably even slows down end-to-end throughput. – jfriend00 May 12 '17 at 19:39
  • @jfriend00 added the code – Sravan May 12 '17 at 19:47
  • The source path will have some sub folders and in these sub folders there will be files. Each file will of size 0.5Mb to 1MB. I am not sure how to restrict the number of files copied in fse.copy operation. I am assuming that nodeJS will try to copy of maximum of 4k files at a time and continue with the next set once this copy is over.. is this not correct ? Also why I am not getting any error message ? – Sravan May 12 '17 at 19:51
  • Your assumption about how either node.js or your own code would deal with too many file operations is apparently not correct. To diagnose, we also need to see the source code for all your `fse.xxxx()` functions you are using. It's very easy to mess up async error handling when using massively nested callbacks like you are doing. For example, you appear to have no error handling on your call to `fse.ensureDir()`. Plus, it looks like you might be expecting `try/catch` to handle exceptions in nested async callbacks which it will not.I would only ever attempt something like this using promises. – jfriend00 May 12 '17 at 20:05
  • The simplest way to control how many of these operations are in-flight at the same time that I know of is to pre-flight the whole operation and build a flat list of things that need to be copied. Then, you can use either something like Bluebird's `Promise.map()` or `async.eachLimit()` on the flat list of things that need to be copied and you can specify the max number of concurrent requests you want to be run. – jfriend00 May 12 '17 at 20:07
  • Thank you for correcting my understanding. I thought of doing the same what you have suggested (getting a list and using async.eachLimit()) but I thought there is some other way. I guess I have to this now.. Will try this. Could you please let me know how can I catch the error with current code ? – Sravan May 12 '17 at 20:15
  • Well `fse.ensureDir()` passes `err` to it's callback and you have no `if (err)` statement to check that. I can't speak for what happens inside of `fse.ensureDir()` or inside of `fse.copy()`. They could have similar issues. – jfriend00 May 12 '17 at 20:18

0 Answers0