Why is the file created from S3 stream empty?

Question

I am trying to access a file in a private S3 bucket from a lambda function identified by Cognito.

Reading the stream works outside a lambda but not inside a lambda
Creating a pre-signed url works inside a lambda
Waiting for the the content to be ready as a string works inside a lambda

I've managed to get a pre-signed url to download the file. Using the same parameters, I've tried to write the read stream to a local file. A file gets created but it's empty. I couldn't catch any error in the process.

const s3 = new AWS.S3({ apiVersion: 'latest' });
const file = 's3Filename.csv'
const userId = event.requestContext.identity.cognitoIdentityId;
const s3Params = {
    Bucket: 'MY_BUCKET',
    Key: `private/${userId}/${file}`, 
};

var fileStream = require('fs').createWriteStream('/path/to/my/file.csv'); 
var s3Stream = s3.getObject(s3Params).createReadStream();

// Try to print s3 stream errors
s3Stream
.on('error', function (err) {
    console.error(err); // prints nothing
});

// Try to print fs errors
s3Stream
.pipe(fileStream)
.on('error', function (err) {
    console.error('File Stream:', err); // prints nothing
})
.on('data', function (chunk) {
    console.log(chunk); // prints nothing
})
.on('end', function () {
    console.log('All the data in the file has been read'); // prints nothing
})
.on('close', function (err) {
    console.log('Stream has been Closed'); // prints nothing
});

I am quite confident that my parameters are correct because I can get a pre-signed url that allows me to download the file.

console.log(s3.getSignedUrl('getObject', s3Params));

I can also read the file content using getObject().promise(). This could work but I'm parsing a CSV file and I'd rather go easy on the memory and parse the stream.

try 
{
    const s3Response = await s3.getObject(s3Params).promise();
    let objectData = s3Response.Body.toString('utf-8'); 
    console.log(objectData);
}
catch (ex) 
{
    console.error(ex);
}

Why is the file created from S3 stream empty? And why is there nothing that prints?

Could it be an access policy issue? If that's the case, why didn't I get any error when executing?

Did you check the following question : https://stackoverflow.com/questions/43799246/s3-getobject-createreadstream-how-to-catch-the-error ? — Nikolay Vetrov, Feb 14 '19 at 21:13
@NikolayVetrov yes, that's the question that lead me to try to catch different events. In particular an error on the s3 read stream. Unfortunately, that didn't help. — 7hibault, Feb 14 '19 at 21:16
Did you try to promisify() with await prefix for the stream method? — Nikolay Vetrov, Feb 14 '19 at 21:18
@NikolayVetrov could you elaborate on that ? I've gotten the response from a getObject promise but not from a stream — 7hibault, Feb 14 '19 at 21:34
Same what you did for s3.getObject. The last code block with a try/catch wrap. — Nikolay Vetrov, Feb 14 '19 at 22:01
@NikolayVetrov `createReadStream` is a synchronous function, why would I do that? I've tried it nevertheless but it ended up failing as a `TypeError: (...) is not a function` — 7hibault, Feb 18 '19 at 08:30

Why is the file created from S3 stream empty?

0 Answers0