0

I have a lot of big files on a remote server and I want to move them into S3. I want to do it at the command line or with a bash script (e.g., I do NOT want to use a gui app like cyberduck) so that I can automate/replicate efforts.

I have tried to mount my remote server onto my local machine using Osxfuse and sshfs and then push it to s3 using s3cmd. This does work but I keep running into errors (connection being lost for no apparent reason; mount errors, etc.).

Is this the best way to do it? Does anyone know a better way to do it?

Thanks, A

ansonw
  • 1,559
  • 1
  • 16
  • 22

2 Answers2

2

You can use minio client aka mc to do the same. 

$ mc cp --recursive localDir/  s3/remoteBucket

In case of network disconnection, mc will provide you an option to resume the upload.

koolhead17
  • 1,944
  • 1
  • 12
  • 20
1

Is your remote server in ec2? Your current setup requires two copies (first to pull data to your local machine via sshfs, then to push to s3 via s3cmd), if you run s3cmd on your remote server directly you can reduce that to one.

If you want to mount s3 as a filesystem, you can also use tools like goofys or s3fs. Again you should do that on your remote server to avoid extra copies.

khc
  • 344
  • 2
  • 8