Download files from S3 bucket + folder

Question

I am writing a Powershell script using the AWS SDK to download a specified amount of files from a particular directory, within a bucket, in S3.

When I run the script, I get this error, for each iteration of the loop:

Read-S3Object : Illegal characters in path.
At line:21 char:5
+     Read-S3Object -BucketName $BucketName -Key $Key -File $LocalFile
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Amazon.PowerShe...dS3ObjectCmdlet:ReadS3ObjectCmdlet) [Read-S3Object], Inv 
   alidOperationException
    + FullyQualifiedErrorId : Amazon.Runtime.AmazonServiceException,Amazon.PowerShell.Cmdlets.S3.ReadS3ObjectCmdlet

I have tried a few variations, I suspected it was an issue with the folder containing : or \

Param( 
   [Parameter(Mandatory=$False)] [String]$WorkingDir = "C:\Temp\Testing\",
   [Parameter(Mandatory=$False)] [String]$BucketName = "BucketName",
   [Parameter(Mandatory=$False)] [String]$DownloadFolder = "Testing",
   [Parameter(Mandatory=$False)] [int]$FileCount = 100
    )

$FilesToDownload = Get-S3Object -BucketName $BucketName -KeyPrefix $DownloadFolder -MaxKey $FileCount

$FilesToDownload | ForEach-Object {

    $Key       = ($_.Key | Out-String)
    $File      = $Key.TrimStart($DownloadFolder + "/")
    $LocalFile = Join-Path $WorkingDir $File

    Read-S3Object -BucketName $BucketName -Key $Key -File $LocalFile

    }

I can get the below to work, which is essentially what the ForEach loop is doing:

Read-S3Object -BucketName BucketName -Key Testing/text.txt -File C:\Temp\Testing\test.txt

Answer 1

Looks like there was some illegal characters in the $LocalFile variable, particularly carriage return, line-feed.

I fixed it by doing:

$LocalFile = $LocalFile -replace "`n",""
$LocalFile = $LocalFile -replace "`r",""

Answer 2

The real problem is the following line.

$Key       = ($_.Key | Out-String)

If you just set $Key = $_.Key then you don't get the carriage return from out-string.

Answer 3

This may be simpler to do:

Get-S3Object -BucketName ${BucketName} -Region ${RegionName} | Select-Object -First ${FileCount} | Copy-S3Object -LocalFolder ${WorkingDir}