216

If I want to download all the contents of a directory on S3 to my local PC, which command should I use cp or sync ?

Any help would be highly appreciated.

For example,

if I want to download all the contents of "this folder" to my desktop, would it look like this ?

 aws s3 sync s3://"myBucket"/"this folder" C:\\Users\Desktop
DJo
  • 2,133
  • 4
  • 30
  • 46
BFlint
  • 2,407
  • 2
  • 16
  • 20

8 Answers8

406

Using aws s3 cp from the AWS Command-Line Interface (CLI) will require the --recursive parameter to copy multiple files.

aws s3 cp --recursive s3://myBucket/dir localdir

The aws s3 sync command will, by default, copy a whole directory. It will only copy new/modified files.

aws s3 sync s3://mybucket/dir localdir

Just experiment to get the result you want.

Documentation:

user2297550
  • 3,142
  • 3
  • 28
  • 39
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • 3
    Took me a few minutes to figure out where to get aws cli. Here it is: https://aws.amazon.com/cli/ – Bjørn Stenfeldt Sep 02 '16 at 21:50
  • 4
    ```aws s3 cp s3://myBucket/dir localdir --recursive```. This works like a charm. If the --recursive flag is skipped, it throws a rather unhelpful error: ```fatal error: An error occurred (404) when calling the HeadObject operation: Key "myBucket" does not exist``` – Siddhartha Apr 22 '19 at 18:43
  • But if the file single zip large size in GBs, what would be the recommandation? – Kanagavelu Sugumar Mar 31 '20 at 12:59
  • 1
    @KanagaveluSugumar Please create a new Question rather than asking via a comment on an old question. – John Rotenstein Mar 31 '20 at 21:51
  • I just want to add: `localdir` is necessary, annoyingly. AWS CLI will only copy the directory contents, not the directory itself. – Mike B Jun 21 '23 at 01:17
14

Just used version 2 of the AWS CLI. For the s3 option, there is also a --dryrun option now to show you what will happen:

aws s3 --dryrun cp s3://bucket/filename /path/to/dest/folder --recursive

Marc Duby
  • 141
  • 1
  • 3
6

In case you need to use another profile, especially cross account. you need to add the profile in the config file 

[profile profileName]
region = us-east-1
role_arn = arn:aws:iam::XXX:role/XXXX
source_profile = default

and then if you are accessing only a single file 

aws s3 cp s3://crossAccountBucket/dir localdir --profile profileName
myPavi
  • 201
  • 2
  • 5
5

In the case you want to download a single file, you can try the following command:

aws s3 cp s3://bucket/filename /path/to/dest/folder
Fabien
  • 4,862
  • 2
  • 19
  • 33
gCoh
  • 2,719
  • 1
  • 22
  • 46
3

You've many options to do that, but the best one is using the AWS CLI.

Here's a walk-through:

  1. Download and install AWS CLI in your machine:

  2. Configure AWS CLI:

enter image description here

Make sure you input valid access and secret keys, which you received when you created the account.

  1. Sync the S3 bucket using:

     aws s3 sync s3://yourbucket/yourfolder /local/path

In the above command, replace the following fields:

  • yourbucket/yourfolder >> your S3 bucket and the folder that you want to download.
  • /local/path >> path in your local system where you want to download all the files.
Darshan Lila
  • 5,772
  • 2
  • 24
  • 34
2

sync method first lists both source and destination paths and copies only differences (name, size etc.).

cp --recursive method lists source path and copies (overwrites) all to the destination path.

If you have possible matches in the destination path, I would suggest sync as one LIST request on the destination path will save you many unnecessary PUT requests - meaning cheaper and possibly faster.

grnc
  • 477
  • 6
  • 13
0

Question: Will aws s3 sync s3://myBucket/this_folder/object_file C:\\Users\Desktop create also the "this_folder" in C:\Users\Desktop? If not, what would be the solution to copy/sync including the folder structure of S3? I mean I have many files in different S3 bucket folders sorted by year, month, day. I would like to copy them locally with the folder structure to be kept.

YAZ84
  • 43
  • 7
0

aws s3 cp s3:// --recursive(if pushing multiple file)

Arul
  • 754
  • 11
  • 21