0

I tried to follow advice provided at https://stackoverflow.com/a/18136205/6608952 but was unsure how to share myAmazonKeypair path in a .pem file on the remote server.

scp -i yourAmazonKeypairPath.pem  fileNameThatYouWantToTransfer.php ec2-user@ec2-00-000-000-15.us-west-2.compute.amazonaws.com:

The command completed after a few minutes with this display:

ssh: connect to host
myBucketEndpointName
port 22: Connection timed out
lost connection

I have a couple of very large files to transfer and would prefer not to have to download the files to my local computer and then re-upload them to the S3 bucket.

Any suggestions?

Community
  • 1
  • 1
Bren
  • 21
  • 1
  • 2
  • Where are files located? – Piyush Patil Jul 19 '16 at 14:32
  • security group - do you open port 22 ? – Frederic Henri Jul 19 '16 at 14:32
  • Nowhere in your question are you using the AWS CLI. It looks like you are asking a question about using `scp` to transfer files to an EC2 server? Please clarify exactly what you are trying to do. – Mark B Jul 19 '16 at 14:59
  • I actually used this syntax: `scp -i myAmazonKeys.csv remote_filename s3_user@bucketname.s3-website-us-east-1.amazonaws.com:` The files are hosted on a website to which I have FTP & SSH access – Bren Jul 19 '16 at 15:30
  • you mixed up S3 and ec2 .. if you want to transfer files to S3 you can't use scope/ftp or any other protocols, as you refer in the questions to upload to s3 you need to use AWS rest API or aws cli tools; you can however upload files to ec2 (as you seemed to indicate in your question by specifying ec2-user@ec2-00-000-000-15.us-west-2.compute.amazonaws.com) if you enable a FTP/SCP server and open the port from the security group – Frederic Henri Jul 19 '16 at 16:17
  • @FrédéricHenri I guess the confusion arose when someone conflated ec2 & S3 with the solution they offered in a comment to the question in this link: http://stackoverflow.com/a/18136205/6608952. I did install and use AWS CLI but found it didn't offer a means to transfer files to an S3 bucket from a remote location. Thanks for your comments. – Bren Jul 20 '16 at 06:51
  • 1
    If you have installed aws cli you can run `aws s3 cp test.txt s3://mybucket/test2.txt` to copy file from your local to S3 bucket or you can run `aws s3 sync . s3://mybucket` to sync all files from current directory to your bucket. check the docs for more example. – Frederic Henri Jul 20 '16 at 07:05
  • Yes - all that is explained in the AWS CLI documentation. And I've successfully tested the cp option on small local files. Problem is my files are 1GB+ in size and located on a remote (webhost) server! – Bren Jul 20 '16 at 07:09
  • can you installed the CLI on the remote server ? – Frederic Henri Jul 20 '16 at 07:12

2 Answers2

5

There is no direct way to upload files to S3 from a remote location. i.e a URL

So to achieve that, you have two options :

  1. Download the file on your local machine and then upload it via AWS Console or AWS CLI.
  2. Download the file in AWS EC2 Instance and upload to S3 by AWS CLI.

The first method is pretty simple, not much explanation needed.

But for the second method, you'll need to do :

  1. Create an EC2 Instance in the same region as the S3 Bucket is. Or if you already have an Instance, then login/ssh to it.
  2. Download the file from the source to the EC2 Instance. via wget or curl whichever is comfortable.
  3. Install AWS CLI on the EC2 Instance.
  4. Create IAM User and Grant him Permission for your S3 Bucket.
  5. Configure your AWS CLI with your IAM Credentials.
  6. Upload your file to S3 Bucket with AWS CLI S3 CP Utility.
  7. Terminate the Instance, if you set up the instance only for this.
Deepak Chaudhary
  • 476
  • 5
  • 15
1

Do it with shell script easily. If you have a list of URLs in files.txt do it like it is described here:

#!/bin/bash 
input="files.txt" 
while IFS= read -r line do   
  name=$(basename "$line")   
  echo $name   
  wget $line   
  aws s3 mv $name <YOUR_S3_URI> 
done < "$input"

Or for one file:

wget <FILE_URL> | aws s3 mv <FILE_NAME> <YOUR_S3_URI>
YolgaAI
  • 86
  • 2