download large files and continue uploading to minio with nodejs, KILLED MY SERVICE?

Question

I have an API built with NodeJS, in that API there is a process where I download a large file using modules request-promise and then it is made a new buffer uploaded to minio. But I have a problem that my API always crashes if the file is above 80-100MB and its NodeJS Killed on the server, how to handle it?

This function is to download the file and convert it into a buffer :

const convertLink = async link => {
  const options = {
    uri: link,
    encoding: null,
    headers: {
      'Content-type': 'application/octet-stream'
    }
  };
  const res = rp.get(options)
    .then((body) => {
      console.log(body)
      const a = new Buffer.from(body);
      return a;
    })
    .catch(err => {
      console.log(err)
      return err;
    });
  return res;
};

this is function for uploading files to minio from miniosdk :

const streamUpload = async (bucketName, objectName, newBuffer) => {
    try {
        const isUploaded = await minioClient.putObject(bucketName, objectName, newBuffer);
        if (isUploaded) {
            return isUploaded;
        }
    } catch (err) {
        return err;
    }
};

score 1 · Answer 1 · answered Mar 03 '20 at 06:49

I think the issue here is you are downloading the file and keeping it in the memory and then uploading it to your minioClient. which is not recommended for large files. you should download that file as a stream and then upload it as a stream too. keeping large files in memory can be the reason to kill your node.js server.

you can try as following example in which I am using request npm library and downloading the file and saving it as a stream to a temporary location and then reading the file from that temporary location and uploading to another URL:-

Downloading file:-

const downloadFile = async (url) => {
    try {

        let tempLocation = "./temp/";
        let fileName="myfile";
        return new Promise((resolve, reject) => {
            request
                .get(url)
                .on('response', function (response) {
                     console.log(response.statusCode) // 200
                     console.log(response.headers['content-type']) 
                })
                .on('error', function (error) {
                    console.log('downloading error', error)
                    reject()
                })
                .on('end', async function () {
                    console.log("donwload finished")
                    resolve();
                })
                .pipe(fs.createWriteStream(tempLocation + '/' + fileName))
        });

    } catch (error) {
        console.log("error in downloadFile", error)
        throw error;
    }
}

now you can upload the file to your minioClient as a stream. you can use fs.createReadStream(file) to read the file as a stream data from that temporary location.

download large files and continue uploading to minio with nodejs, KILLED MY SERVICE?

1 Answers1