2

I need to download the directory tree of a folder in a bucket recursively, with the caveat of not downloading any of the files themselves (they are large). This is what I have so far:

s3cmd --dry-run --recursive --no-delete-removed --verbose --rinclude='^[^.]*$' 
    --rexclude='[^.]*$' sync s3://bucket_name/folder/ ~/Downloads/local_folder/

which correctly skips the files, but it isn't downloading the folders.

Here are some related questions I found

Community
  • 1
  • 1
M.R.
  • 1,053
  • 2
  • 13
  • 30
  • As noted in the answer to the other question you cited, there are no folders to download. The folders in S3 are largely an illusion, a side effect of the `/` delimiters in the key names... s3cmd may not have a simple way of downloading things that don't actually exist. A direct API call to the REST interface for ListObjects allows you to fetch "common prefixes" -- the "folders" -- but whether s3cmd leverages this in a useful way is unknown to me, fwiw. – Michael - sqlbot Oct 07 '15 at 02:36

1 Answers1

3

Amazon S3 does not have folders. The closest concept is a CommonPrefix, which allows listing of files "within" a directory.

You would have to write your own program (eg in Python using boto) to retrieve these common prefixes and create a directory structure locally.

BTW, these days it is better to use the AWS Command-Line Interface (CLI) rather than s3cmd since it can work with all AWS services rather than just Amazon S3.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470