Copy Amazon S3 Bucket Contents to Local Folder using PowerShell Script

Question

I found a PowerShell script to copy all files from an Amazon S3 bucket to a local folder on one of our on-site servers:

# Your account access key - must have read access to your S3 Bucket
$accessKey = "MyAccessKey"
# Your account secret access key
$secretKey = "MySecretKey"
# The region associated with your bucket e.g. eu-west-1, us-east-1 etc. (see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions)
$region = "eu-west-1"
# The name of your S3 Bucket
$bucket = "MyBucket/"
# The folder in your bucket to copy, including trailing slash. Leave blank to copy the entire bucket
$keyPrefix = ""
# The local file path where files should be copied
$localPath = "C:\test\"   

$objects = Get-S3Object -BucketName $bucket -KeyPrefix $keyPrefix -AccessKey $accessKey -SecretKey $secretKey -Region $region

foreach($object in $objects) {
$localFileName = $object.Key -replace $keyPrefix, ''
if ($localFileName -ne '') {
    $localFilePath = Join-Path $localPath $localFileName
    Copy-S3Object -BucketName $bucket -Key $object.Key -LocalFile $localFilePath -AccessKey $accessKey -SecretKey $secretKey -Region $region
}
}

If I run the first part on it's own (up to and including the Get-S3Object line) and then display the $objects variable it displays the details of the files one after the other with the below info in PowerShell

ETag : "38763873d83763c3876"
BucketName : MyBucketName
Key : 3287653_32876_to_38763_3876_client.xml
LastModified : 14/10/2016 11:26:51
Owner : Amazon.S3.Model.Owner
Size : 485
StorageClass : Standard

So it's successfully getting the files from the bucket... However, if I run the whole script I get an error for each loop iteration which says:

Copy-S3Object : The specified key does not exist.
At line:5 char:9
+         Copy-S3Object -BucketName $bucket -Key $object.Key -LocalFile ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (Amazon.PowerShe...yS3ObjectCmdlet:CopyS3ObjectCmdlet) [Copy-S3Object]
, InvalidOperationException
+ FullyQualifiedErrorId : Amazon.S3.AmazonS3Exception,Amazon.PowerShell.Cmdlets.S3.CopyS3ObjectCmdlet

I therefore suspect it's something to do with the command or variables from the loop...

foreach($object in $objects) {
$localFileName = $object.Key -replace $keyPrefix, ''
if ($localFileName -ne '') {
    $localFilePath = Join-Path $localPath $localFileName
    Copy-S3Object -BucketName $bucket -Key $object.Key -LocalFile $localFilePath -AccessKey $accessKey -SecretKey $secretKey -Region $region
}
}

...but have not been able to figure out what.

Does anyone have any bright ideas on what I'm doing wrong please?

score 3 · Answer 1 · edited Nov 12 '19 at 19:33

3

The problem is, that you are trying to use Copy-S3Object on something that is not an object.

What you perceive as 'folders' in S3 are really not folders, but just prefixes of the object name.

I've had success with a construct such as:

Get-S3Object -BucketName $bucket -KeyPrefix $key |
    Copy-S3Object -LocalFolder $localPath

edited Nov 12 '19 at 19:33

henrycarteruk

12,708
2
36
40

answered Sep 01 '17 at 09:30

Emil

432
4
11

henrycarteruk · Answer 2 · 2016-11-17T13:55:14.120

1

You can use the AWSCLI commands for S3 in Powershell, they should be already be installed as they come with AWS Powershell Cmdlets.

It's also a much simpler command to understand:

aws s3 cp s3://bucketname C:\folder --recursive

edited Nov 17 '16 at 13:55

answered Nov 17 '16 at 13:47

henrycarteruk

12,708
2
36
40

1

Note this won't work on windows if the files contain reserved filename characters. The folders will be created but not files and there is no error message in a command prompt. – Jeremy Thompson Oct 27 '19 at 02:57

Copy Amazon S3 Bucket Contents to Local Folder using PowerShell Script

2 Answers2