Connect to AWS S3 bucket from Nodejs app ( persistent connection required)

Question

I have nodejs/express app from which I want to connect to AWS S3.

I do have a temporary approach to make connection,

environment file

aws_access_key_id=XXX
aws_secret_access_key=XXXX/
aws_session_token=xxxxxxxxxxxxxxxxxxxxxxxxxx

S3-connection-service.js

const AWS = require("aws-sdk");

AWS.config.update({
    accessKeyId: `${process.env.aws_access_key_id}`,
    secretAccessKey: `${process.env.aws_secret_access_key}`,
    sessionToken: `${process.env.aws_session_token}`,
    region: `${process.env.LOCAL_AWS_REGION}`
});

const S3 = new AWS.S3();

module.exports = {
    listBucketContent: (filePath) =>
        new Promise((resolve, reject) => {
            const params = { Bucket: bucketName, Prefix: filePath };
            S3.listObjects(params, (err, objects) => {
                if (err) {
                    reject(err);
                } else {
                    resolve(objects);
                }
            });
        }),
       ....
       ....
 }

controller.js

   const fetchFile = require("../../S3-connection-service.js");

   const AWSFolder = await fetchFile.listBucketContent(filePath);

Fine it's works and I'm able to access S3 bucket files and play with it.

PROBLEM

The problem is connection is not persistent. Since, I use session_token, connection remains alive for sometime and again after sometime new tokens will be generated, I have to copy-paste them in env file and re-run the node app.

I really have no idea how can I make connection persistent ?

Where to store AWS confidential/secrets and how to use them to connect to S3 so connection remains alive ?

Where and how is that app running? – kichik Jan 02 '23 at 06:18 — kichik, Jan 02 '23 at 06:18

score 1 · Accepted Answer · answered Dec 29 '22 at 11:26

Just remove

AWS.config.update({
    accessKeyId: `${process.env.aws_access_key_id}`,
    secretAccessKey: `${process.env.aws_secret_access_key}`,
    sessionToken: `${process.env.aws_session_token}`,
    region: `${process.env.LOCAL_AWS_REGION}`
});

code block from lambda source in file S3-connection-service.js

Attach a role to lambda function with proper permissions. You will have same functionally.

For local development.

You can set environment variable before testing your application.

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-west-2

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

If you are using any IDE you can set these environment variables on it.

If you are testing from cli

$ AWS_ACCESS_KEY_ID=EXAMPLE AWS_SECRET_ACCESS_KEY=EXAMPLEKEY AWS_DEFAULT_REGION=us-west-2 npm start

Thanks for the answer. I just created a role and using its accesskey and secretkey to get records from S3 bucket which is working fine. I still have doubt on the lambda function. How can I attach a role to lambda function ? fyi, I'm still using AWS.config.update({...}) but now I don't use `sessionToken` in it. — micronyks, Jan 02 '23 at 10:50
This video help you to attach/change lambda role https://youtu.be/T7cny-uyCuA first part creation of role second part attach it to lambda — Mehmet Güngören, Jan 02 '23 at 11:21

score 0 · Answer 2 · answered Dec 23 '22 at 05:52

0

connect to S3 so connection remains alive ?

You can't make one request to S3 and keep it alive forever.

These are your options:

Add a try/catch statement inside your code to handle credentials expired error. Then, generate new credentials and re-initialize the S3 client.
Instead of using a Role, use a User. (IAM Identities). User credentials can be valid forever. You won't need to update the credentials in this case.
Do not provide the credentials to AWS.config.update like you are doing right now. If you don't provide the credentials, the AWS client will try to read them from your ~/.aws/credentials file automatically. If you create a script to update them every hour (ex: a cronjob), then your credentials will be up-to-date at all times.

answered Dec 23 '22 at 05:52

Brian

1,056
9
15

I have try/catch statement and I'm informing the user about token authentication failed. – micronyks Dec 23 '22 at 06:28
Thanks for replying but it doesn't move me anywhere... – micronyks Dec 23 '22 at 12:37
"I'm informing the user about token authentication failed" -> Who is the user? How can the user refresh the credentials? What are the issues with the 2nd and 3rd options that they don't work for you? – Brian Dec 23 '22 at 12:59
There is no issue. I really have no idea how to proceed further. I have AWS login with me. I'll login and copy the tokens, change them in the application when it gets expired but as of now it is a manual process. I want to get rid of it and want to automate it. I really don't know how to proceed. Any reference link would also help me. – micronyks Dec 25 '22 at 12:04
Where is your express app running? Is it on EC2? – Brian Dec 25 '22 at 12:32
I use Lambda + API gateway. My node/express code is within Lambda function. Lambda communicates with mysql to fetch data. – micronyks Dec 26 '22 at 05:26
In that case you don't need to provide credentials at all. Set up a role for your Lambda with the required permissions and delete the AWS.config.update line and it will work. See: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-lambda.html – Brian Dec 26 '22 at 05:44
Thanks for the reply. but how can I handle it at development time. At dev time, I run node/express independently and from node, I want to connect to S3 bucket. – micronyks Dec 26 '22 at 06:29
Can you explain a little bit more? I don't understand the concern. Are you asking about how to set up development and production stages? – Brian Dec 26 '22 at 08:32

Connect to AWS S3 bucket from Nodejs app ( persistent connection required)

2 Answers2