Node.js - read and download all files in directory from server and save locally

Question

I have a Node Webkit Desktop App and need to download files from the server and save locally for when users are offline. I can download and save a file when I know what the file name is, but how do I read the contents of a directory on the server so I can download each file?

function cacheFiles(filelink, filepath, cb) {
    var path_array =  filelink.split("/");
    var foldername =  path_array[path_array.length - 2]

//create new folder for locally html files
var newdir = filepath + '/' + foldername;
if (fs.existsSync(newdir)){
    alert('file already exists, cannot cache this file.');
} else {
    fs.mkdirSync(newdir);
}
//download and save index.html - THIS WORKS
var indexfile = fs.createWriteStream(newdir+'/index.html');
var request = http.get(filelink, function(response) {
    response.pipe(indexfile);
    indexfile.on('finish', function() {
        indexfile.close(cb);
    });
});
//read contents of data folder - THIS DOESN'T WORK
var datafolder = filelink.replace('index.html','');

fs.readdir( datafolder, function (err, datafiles) { 
    if (!err) {
        console.log(datafiles);
    }else{
        console.log(err) ; 
    }   
});

}

The error I get in my console is:

"ENOENT: no such file or directory, scandir 'C:\Users\my.name\desktopApp\app\http:\www.mysite.co.uk\wp-content\uploads\wifi_corp2\data'"

The above is looking for the files locally and not at the online link I supplied in filelink eg. http://www.mysite.co.uk/wp-content/uploads/wifi_corp2/data

on my server eg. http:\www.mysite.co.uk\wp-content\uploads\wifi_corp2-1475754605\data — LeeTee, Oct 07 '16 at 14:02
Can you access the list of files, the list of filepaths to download? — Martin Gottweis, Oct 07 '16 at 14:04
request http folder will output an html file. Type your folder path in your browser and parse this html file. (depend on your server configuration) — Steeve Pitis, Oct 07 '16 at 14:05
yes I can access the list of files in my browser. I get an error in the console. see amended query above — LeeTee, Oct 07 '16 at 14:08
I didn't downvote, but it is likely because the question is a bit hard to understand, and it doesn't seem easy to _track down_ where the issue might be coming from. — John Weisz, Oct 07 '16 at 15:57
what is the full path of you folder where you saved files locally? Did you try to add en '\\' at the end of the datafolder variable in readdir call? — Supersharp, Oct 10 '16 at 21:51

cchamberlain · Answer 1 · 2016-10-11T05:25:57.093

2

The following code doesn't read a remote file system, it's used for reading files on your local hard drive.

import fs from 'fs'
import path from 'path'

fs.readdir(path.resolve(__dirname, '..', 'public'), 'utf8', (err, files) => {
    files.forEach((file) => console.info(file))
})

Will print out all the file names from one directory up and in a 'public' directory from the script location. You can use fs.readFile to read the contents of each file. If they are JSON, you may read them as utf8 strings and parse them with JSON.parse.

To read files from a remote server, they must be served via express or some other static file server:

import express from 'express'
const app = express()
app.use(express.static('public'))
app.listen(8000)

Then on the client end you could use fetch or request http library to call the express endpoint hosted at port 8000 (in this simple example.

edited Oct 11 '16 at 05:25

answered Oct 11 '16 at 05:20

cchamberlain

17,444
7
59
72

Hi, thanks for your help.Any chance you could elaborate on how to do this exactly? I have just installed express within my existing node webkit app and added the 4 lines of code you have displayed above. I now get an error, "unexpected token import" Can you assist further as I am totally lost. many thanks – LeeTee Oct 11 '16 at 08:25
I amended, "import express from 'express'" to "const express = require("express");" but now what? How do I read the files in the directory? Thanks – LeeTee Oct 11 '16 at 09:10
This is es6 syntax. It can be transpiled with babel. – cchamberlain Oct 11 '16 at 14:07
You can't do that remotely. You would need to use readdir on the server and expose an API metadata endpoint that describes where all the files live, then call the metadata endpoint with an http client to find the locations. – cchamberlain Oct 11 '16 at 14:11
If this is not your server, you're out of luck. – cchamberlain Oct 11 '16 at 14:11

score 1 · Answer 2 · answered Oct 11 '16 at 05:34

1

You are mixing up the server code with the desktop app code. Obviously the desktop app can't do a readdir on your server fiiles. Just install a backup or download plugin on Wordpress.

answered Oct 11 '16 at 05:34

Jason Livesay

6,317
3
25
31

Not sure what you mean by backup or plugin on wordpress. To do what exactly? – LeeTee Oct 11 '16 at 09:06

score 0 · Accepted Answer · answered Oct 11 '16 at 09:17

0

OK, Im thinking the best way around this is to use Ajax to call a PHP function on the server to read the contents of the file.

answered Oct 11 '16 at 09:17

LeeTee

6,401
16
79
139

Yes this is essentially the same as using an AJAX call to a nodejs endpoint that reads the filesystem and returns the results. Given the question mentions Node, that's the language I used. – cchamberlain Oct 11 '16 at 18:10

Node.js - read and download all files in directory from server and save locally

3 Answers3