0

I have tens of thousands of directories with a few files in each one on a non-AWS VPS (approx 1TB of data). I want to move them all to S3.

I can either zip these in to chunks of 7GB, move (wget or whatever) to EC2 (8GB Ubuntu), unzip and s3cmd them to S3.

OR

go straight to S3 from my VPS with s3cmd sync directory?

which method would be best for performance and reliability?

Thanks

Gordon
  • 393
  • 1
  • 3
  • 8

1 Answers1

2

There are two factors that would make the decision for me.

  1. What is the avg file size? (Thousands of little files can take more time than several large files)
  2. What kind of compression can you get?

If you decide to use an intermediary instance, you can attach a 1 TB EBS volume to handle the files while transferring. It will add a bit of cost, but you wont need to keep the volume once you are done.

datasage
  • 19,153
  • 2
  • 48
  • 54
  • Awesome, thanks for the heads up about EBS. AWS is an amazing "thing", despite not really knowing or utilizing it's full potential. I failed to add that there are about 3.5 million small (few MB) files. – Gordon Mar 19 '13 at 16:18