Programmatically move files from an S3 bucket to a Windows EC2 and vice-versa

Question

I want to push files to a Windows EC2 and also get folders from my EC2 and store it in S3. As I have many EC2s, I wanted to automate this - I currently have a Node.js server on ECS that uses ssh2-sftp-client library. With the following piece of code I'm able to push files into it (similarly, I'm able to extract files from it and upload it to S3):

            var Client = require('ssh2-sftp-client');          
            var sftp = new Client(newClientID);
            sftp.connect({
                host: host,
                username: 'username',
                password: 'password',
                port: '22',
                tryKeyboard: true
            }).then(async () => {
                try {
                    if (file.filename && file.file) {
                        await sftp.put(file.file, `C:/Users/user/Desktop/${file.filename}`);
                        console.log(`Successfully pushed ${file.filename}`);
                    }

                    sftp.on('error', error => {
                        console.log(error);
                        sftp.end();
                    });

                    resolve();
                } catch (ex) {
                    console.log("SFTP EXCEPTION PUSHING FILES TO INSTANCE", ex);
                } finally {
                    sftp.end();
                }
            }

But this is not a robust solution - even when I have 5 or 6 users trying to push at the same time the server errors because it has too many active SSH connections.

Is there a better way to do this? All I want to do is upload/download specific directories using Node.js (for a Windows EC2).

You might want to "pull" from S3 on a regular cycle, on each machine using the [AWS Command-Line Interface (CLI)](http://aws.amazon.com/cli/) `aws s3 sync` command. See: [sync — AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) — John Rotenstein, Apr 30 '20 at 03:01
@jarmod thank you! SSM Run Command is exactly what I needed. — SI2, May 03 '20 at 03:34

jarmod · Accepted Answer · 2020-05-03T15:17:21.723

1

One good way to automate the running of scripts on EC2 is to use SSM Run Command. If you set up each EC2 instance correctly, then it becomes a managed instance and you can trigger the running of scripts across a fleet of EC2 instances, chosen by tags, for example.

edited May 03 '20 at 15:17

answered May 03 '20 at 14:10

jarmod

71,565
16
115
122

Programmatically move files from an S3 bucket to a Windows EC2 and vice-versa

1 Answers1