iterating through S3 bucket with stream wrapper

Question

I am trying to list all the items in my Amazon S3 bucket. I have several nested directories in it.

dir1/
dir1/subdir1/
dir1/subdir2/
dir1/subdir3/
dir2/
dir2/subdir1/
dir2/subdir2/
...

Each subdirectory contains several files. I need to get a nested array with this file structure.

I'm using the Amazon AWS SDK for PHP 2.4.2

This is my code:

$dir = 's3://bucketname';

$iterator = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($dir));

foreach ($iterator as $file) {
    echo $file->getType() . ': ' . $file . "\n";
}

However, the result only lists files lying in the bucket, not files lying in directories/subdirectories (files with prefixes) or the directories itself.

If I iterate through ($dir.'/folder') there is no result at all.

I I pass RecursiveIteratorIterator::SELF_FIRST as the second argument to the constructor of the iterator, I get only first level directories – no subdirectories.

How can I use the AWS stream wrapper and the PHP RecursiveIterator to list all the files in all the directories in my bucket?

I hope someone can help me.

Thank you!

See if http://stackoverflow.com/a/4165046/208809 solves your problem, e.g. pass `RecursiveIteratorIterator::SELF_FIRST` as the second argument to the constructor of the `RecursiveIteratorIterator`. If it does, consider deleting your question. If it doesn't [edit] your question to point out that it's not an issue with the iteration mode. — Gordon, Aug 11 '13 at 12:04
Thank you @Gordon. Tried it, didn't change the problem itself (no subdirectories or files in subdirectories shown). I edited the question. — basbebe, Aug 12 '13 at 13:27
Does the actual name of the bucket you are listing contain ".mydomain.com"? If the bucket name is actually something like "bucketname", then you must only use "s3://bucketname" in the $dir variable. — Michael Dowling, Aug 12 '13 at 17:16
@MichaelDowling: Yes, the actual name of the bucket contains my domain.com – I edited the post accordingly, thanks! — basbebe, Aug 13 '13 at 15:14
We just made an SDK release last night. Can you check to see if it resolved your issue? — Michael Dowling, Aug 13 '13 at 22:41
@MichaelDowling: I just checked with the new release (2.4.3): No change. If I iterate through a folder on my server it works as expected. Just not with the S3 stream wrapper. — basbebe, Aug 14 '13 at 10:39

score 2 · Accepted Answer · answered Feb 17 '14 at 18:49

I had the same problem, and I solved this using:

use \Aws\S3\StreamWrapper;
use \Aws\S3\S3Client;

private $files = array();
private $s3path = 'YOUR_BUCKET';
private $s3key = 'YOUR_KEY';
private $s3auth = 'YOUR_AUTH_CODE';

public function recursive($path)
{
    $dirHandle = scandir($path);

    foreach($dirHandle as $file)
    {
        if(is_dir($path.$file."/") && $file != '.' && $file != '..')
        {
            $this->recursive($path.$file."/");
        }
        else
        {
           $this->files[$path.$file] = $path.$file;
        }
    }
}

public function registerS3()
{
    $client = S3Client::factory(array(
        'key'    => $this->s3key,
        'secret' => $this->s3auth
    ));

    $wp = new StreamWrapper();
    $wp->register($client);
}

public function run()
{
    $folder = 's3://'.$this->s3path.'/';

    $this->registerS3();
    $this->recursive($folder);
}

Now, if you do a DUMP in $this->files, should show all the files on the bucket.

iterating through S3 bucket with stream wrapper

1 Answers1