NodeJS write file to AWS S3 - Promise.All with async/await not waiting

Question

I writing a number of files to AWS S3 using NodeJS. I have the data in a stringifed variable but have also tested reading in a file. The files write successfully to S3 so it does work but the "await Promise.all()" completes before the file uploads complete. I need to know when the files have completed uploading.

    // Stream File Data to S3
    const data = "blah blah blah"
    const s3Path = "my_path"
    const promises = []
    promises.push(uploadFile(data, `${s3Path}/1111.xml`))
    promises.push(uploadFile(data, `${s3Path}/2222.xml`))
    promises.push(uploadFile(data, `${s3Path}/3333.xml`))
    promises.push(uploadFile(data, `${s3Path}/4444.xml`))
    const completed = await Promise.all(promises)

below is the function I'm using to upload the files.

const uploadFile = (data, key) => {

     // const fileContent = data
    const fileContent = fs.readFileSync("/full_path/test.xml")

    // Setting up S3 upload parameters
    const params = {
        "Bucket": "my_bucket",
        "Key": key,
        "ContentType": "application/xml",
        "Body": fileContent
    }

    // Uploading files to the bucket
    s3.upload(params, (err, data2) => {
        if (err) {
            throw err
        }
        console.log(`File uploaded successfully. ${data2.Location}`)
        return true
    })
}

Is there a way to update this so promise.all() waits for all uploads to completed before progressing?

thankyou

jfriend00 · Accepted Answer · 2021-02-27T23:43:21.073

That's because uploadFile() does not return a promise. And, Promise.all() only does something useful when you pass it an array of promises that are tied to their asynchronous operations.

I suspect that s3.upload() has a promise interface that you should be using.

const uploadFile = async (data, key) => {

    const fileContent = await fs.promises.readFile("/full_path/test.xml");

    // Setting up S3 upload parameters
    const params = {
        "Bucket": "my_bucket",
        "Key": key,
        "ContentType": "application/xml",
        "Body": fileContent
    }

    // Uploading files to the bucket
    const data2 = await s3.upload(params).promise();
    console.log(`File uploaded successfully. ${data2.Location}`);
    return true;
}

Now uploadFile() returns a promise that resolves when the asynchronous operations in it are done (or rejects is there's an error). So, then when you push the results of uploadFile() to an array and pass that to Promise.all(), it will be able to tell you when they are all done.

FYI, if uploadFile() always reads the same fileContent every time, then you ought to read that once outside the function and perhaps pass the data into uploadFile().

FYI, here's some AWS doc on using promises in the Javascript interface. Worth reading.

NodeJS write file to AWS S3 - Promise.All with async/await not waiting

1 Answers1

Linked