4

The below script copies all files from a folder structure then pastes them to an S3 bucket. However I want it to be able to skip files that have not been changed since the last upload, to avoid duplicating the upload. Does anyone know how I can get a if file exists check or last modified? 

Import-Module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"
$bucket="bucketname"
$source="e:\dfs\*"
$outputpath="C:\temp\log.txt"
$AKey="xxxx"
$SKey="xxxx"

Set-AWSCredentials -AccessKey $AKey -SecretKey $SKey -StoreAs For_Move
Initialize-AWSDefaults -ProfileName For_Move -Region eu-west-1

Start-Transcript -path $outputpath -Force
foreach ($i in Get-ChildItem $source -include *.* -recurse)
{
    if ($i.CreationTime -lt ($(Get-Date).AddDays(-0)))
    {
        $fileName = (Get-ChildItem $i).Name
        $parentFolderName = Split-Path $i -Parent

        Write-S3Object -BucketName $bucket -Key dfs/$parentFolderName/$filename -File $i
    }
}
Anthony Neace
  • 25,013
  • 7
  • 114
  • 129
Geraint Williams
  • 41
  • 1
  • 2

2 Answers2

3

For a very basic "does the file exist yet?" you could use Get-S3Object with the same location, and test each file before attempting to upload it.

if (!(Get-S3Object -BucketName $bucket -Key dfs/$parentFolderName/$filename)) {
    Write-S3Object -BucketName $bucket -Key dfs/$parentFolderName/$filename -File $i
}

Comparing the Changed date to the Last Upload is slightly more complex, but you can use a test more like:

$localModified = (Get-ItemProperty -Path $fileName).LastWriteTime
$s3Modified = (Get-S3Object -BucketName $bucket -Key $file -Region us-east-1).LastModified | Get-Date

if ($s3Modified -lt $localModified) {
    Write-S3Object -BucketName $bucket -Key dfs/$parentFolderName/$filename -File $i
}
James Ruskin
  • 808
  • 7
  • 13
  • Thank you! Issue I now have is that objects uploaded to S3 get their date changed so comparing them like this fails. Maybe S3 isnt the best backup solution like this. Considered maybe tagging the S3 object metadata with the time stamp. Seems overly complicated though – Geraint Williams May 29 '15 at 19:44
  • 1
    When you say "get their date changed", what do you mean? Even if the date of the upload is higher than the last local write time, it's still an indication that it needs to be uploaded again, right? – James Ruskin May 29 '15 at 21:12
1

Putting those together I got the following:

Import-Module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"
$bucket="<my bucket name>"
$source="C:\dev\project\*"
$outputpath="C:\dev\log.txt"
$AKey="<key>"
$SKey="<secret>"
$region="<my AWS region>"

Set-AWSCredentials -AccessKey $AKey -SecretKey $SKey -StoreAs For_Move
Initialize-AWSDefaults -ProfileName For_Move -Region $region

Start-Transcript -path $outputpath -Force
foreach ($i in Get-ChildItem $source -include *.* -recurse)
{
    if ($i.CreationTime -lt ($(Get-Date).AddDays(-0)))
    {
        $fileName = (Get-ChildItem $i).Name
        $parentFolderName = Split-Path $i -Parent
        $key = "$i"

        $localModified = (Get-ItemProperty -Path $i).LastWriteTime
        $remoteObject = Get-S3Object -BucketName $bucket -Key $key -Region $region
       if($remoteObject -eq $null) {
           Write-S3Object -BucketName $bucket -Key $key -File $i
           "Added new file $i"
       } else {

           $s3Modified = $remoteObject.LastModified | Get-Date

           if ($s3Modified -lt $localModified) {
               Write-S3Object -BucketName $bucket -Key $key -File $i
               "Updated $i"
           }
       }
   }
}

Note: this is the first Powershell script that I've ever written - so forgive me if I have the style and approach wrong.

brendan
  • 290
  • 2
  • 13