0

I am writing PHP script for uploading folders if not found in amazon S3 bucket using s3cmd . For that first i need to get all folder names and store it in array and then check array using array_search if folder name is there or not and proceed to next step .

I used get_object_list and list_objects in PHP both will not return only folder names . So i wonder is there any way to do it .

John Saunders
  • 160,644
  • 26
  • 247
  • 397
maheshiv
  • 1,778
  • 2
  • 17
  • 21
  • S3 objects look like they have folders, but they don't really exist. You can look for a *prefix*. If this doesn't help, please list the commands you are trying and some examples of what you want. – tedder42 Jan 17 '15 at 21:39
  • Unlike forum sites, we don't use "Thanks", or "Any help appreciated", or signatures on [so]. See "[Should 'Hi', 'thanks,' taglines, and salutations be removed from posts?](http://meta.stackexchange.com/questions/2950/should-hi-thanks-taglines-and-salutations-be-removed-from-posts). – John Saunders Jan 18 '15 at 01:41

2 Answers2

0

Amazon S3 uses is "flat" storage system. It does not actually support folders/sub-directories. Rather, the sub-directory path is stored as part of the filename, known as the "key".

For example, foo.txt stored in a folder called bar actually creates an object with a key name of bar/foo.txt. When retrieving the object it appears as though it is in the bar directory, but it really isn't.

S3 does support the ability to retrieve a list of Common Prefixes that is effectively a list of directories, but there is no guarantee that such prefixes always exist, especially for empty directories.

Rather than writing a script to "upload folders if not found", take a look at the AWS Command-Line Interface (CLI). It includes a command aws s3 sync that can automatically copy files between locations, including only copying new or modified files. That might totally avoid the need for you to write your own script.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
0

Amazon S3 is a cloud/object storage with a flat namespace. There is not concept of a folder or directory on Amazon S3. What you do have is a key and its CommonPrefixes

To get a similar feeling of POSIX one has use ListObjects() with special delimiters to provide an emulation for file system browsing.

https://github.com/minio/mc is built with such a purpose what you want is fairly easy to do in mc.

Here is an example approach on how you can see this working. mc features progress bar, session management, script friendly --json output and many more. 

$ mc --json ls https://s3.amazonaws.com/miniocloud | jq 'if .type == "folder" then .name else empty end'
"G/"
"Go/"
"asdfasdfadsf/"
"bin/"
"ema/"
"emacs/"
"new/"
"te1231231231231231231231/"
"test/"
Harshavardhana
  • 1,400
  • 8
  • 17