15

Here is answer about how move files from one s3 bucket to another using sync command from command line (cli):

aws s3 sync s3://from_my_bucket s3://to_my_other_bucket

Can this be invoked from javascript sdk? With searching I did not find any sync named method. So is it supported in sdk?

Cherry
  • 31,309
  • 66
  • 224
  • 364

3 Answers3

1

You can see how this node package does it.

Basically, it makes use of several S3 instance methods to sync two buckets.

Noel Llevares
  • 15,018
  • 3
  • 57
  • 81
0

I needed to sync thousands of files, and decided to make use of AWS Data Pipeline to do it. I update the pipeline definition to point to the correct source and destination folders from my Lambda function, and then invoke it.

It uses 2 S3DataNodes and a CopyActivity.

jfrumar
  • 1,970
  • 1
  • 12
  • 7
0

I found this really helpful article on cloning S3 bucket using Node.JS package "aws-sdk": [Node Clone S3 Bucket by Rajesh Babu][1]

I am quoting his approach here:

  • Fetch the list of Keys from the bucket and the target Prefix. (check AWS-SDK Javascript APIs)
  • Separate the files and directories, because we clone the directories and download the files.
  • Clone all the directories first, and then move on to download the files.
  • Download the files through streams and log success and failure respectively.
[1]: https://blog.bitsrc.io/a-practical-guide-to-building-a-node-js-service-on-an-aws-s3-bucket-aff19105ba83
Jatin Verma
  • 53
  • 5