How do you Write an S3 Object using AWS Tools in powershell and verify using an md5checksum

Question

I want to use the integrity features of AWS specified in: https://aws.amazon.com/premiumsupport/knowledge-center/data-integrity-s3/

to make md5checksums work. However, doing this:

Write-S3Object -BucketName BUCKETNAME -File .\test.txt -Key another_key4 -Metadata @{ md5checksum = 'asdfasdfasdfsa'} -HeaderCollection @{ md5checksum = 'asdfasdfasdfsa'} -Verbose

does not seem work as the operation completes successfully and uploads the file on S3. I want to fail as the md5checksum is not 'asdfasdfasdfsa'

When I look in the console, I can see that my -Metadata did at least something as:

How can I make so that the Write-S3Object operation fails unless I give it the correct md5checksum?

Looks like you cannot do that with powershell natively - https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-object.html has a `--content-md5` option that powershell does not offer. Maybe try the `Content-MD5` header. — luk2302, Mar 12 '21 at 12:27
This thread has some info and multipart upload cases as well — TheGameiswar, Mar 12 '21 at 12:34
If you put that as an answer @TheGameiswar then i will approve! — Nick, Mar 12 '21 at 12:57

score 0 · Answer 1 · answered Mar 12 '21 at 13:08

Below is based on https://forums.aws.amazon.com/message.jspa?messageID=414257 , pasting the entire (untested) code,so to avoid link riot

#cleanup, zip, and push to s3
#The folders are organized to limit my need to upload big/long changes to s3
#Install is tiny so I can customize the install script quickly
#Config contains customization that is large, but stable. Skip if no changes.
#Package contains all the permanent zips and packages from vendors, very stable, but almost 1 GB so it takes 15 minutes to upload. I want to skip that upload if the file hasn't changed.
 
$magicSource = 'MagicSource'
$scriptPath = Split-Path -Parent $PSCommandPath  # Get the location of this script.
 
$bucket = 'magic-xpi41-source-bucket'
if ((Test-S3Bucket -BucketName $bucket) -eq $false ){
    Write-Output "Missing $bucket or you do not have access."
}
 
function Copy-ZipS3 {
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided'),
[string] $FileName = $(throw 'Error: A local filename must be provided')
)
    if ( (Test-S3CRC -BucketName $BucketName -Key $Key -File $FileName) -eq 'ok' ){
        Write-Output 'Skipping upload. File matches.'
        return
    }
    try {
        $timeS3 = Measure-Command {
            Write-S3Object -BucketName $BucketName -Key $Key -File $FileName
        }
        Write-Output "Uploaded $filename to $bucketname -> $Key.`nTime to upload to S3: $timeS3"
    }
    catch {
        $last_error = $Error[0]
        "`nError/Exception:`nKey: $key`nFilename: $filename`n$last_error`n"
    }
}
 
function Get-Md5Local {
param(
[string] $FileName = $(throw 'Error: A local filename must be provided')
)
    $md5 = New-Object System.Security.Cryptography.MD5CryptoServiceProvider
    $hash = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($FileName))).Replace('-', '').ToLower()
    return $hash
}
 
function Get-Md5S3 {
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided')
)
    $etag = (Get-S3Object -BucketName $BucketName -Key $Key)[0].ETag
    $etag = $etag.Trim('"') # remove quotes from etag
    return $etag
}
 
 
function Test-S3CRC {
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided'),
[string] $FileName = $(throw 'Error: A local filename must be provided'),
[switch] $Verbose = $false
)
    $hash = Get-Md5Local -FileName $Filename
    $etag = Get-Md5S3 -BucketName $BucketName -Key $Key
    if ($Verbose) {"MD5 verified: $FileName ($hash) -> $BucketName : $Key ($etag)" }
 
    if ($etag -ne $hash)
    {
        return "Tags are not equal:`nHash = $hash`nEtag = $etag" 
    }
    else{ 
        return 'ok' 
    }
}
 
function Repair-S3ETagOld{
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided')
)
    if (Get-S3Object -BucketName $BucketName -Key $Key){
        # to get the right etag, we must copy object back to itself
        $message = Copy-S3Object -SourceBucket $BucketName -SourceKey $Key -DestinationKey "$Key.copy"
        $message = Copy-S3Object -SourceBucket $BucketName -SourceKey "$Key.copy" -DestinationKey $Key
        $message = Remove-S3Object -BucketName $BucketName -Key "$Key.copy" -Force
    }
}
 
function Repair-S3ETag{
param(
[string] $BucketName = $(throw 'Error: An object key must be provided'),
[string] $Key = $(throw 'Error: An object key must be provided')
)
    if (Get-S3Object -BucketName $BucketName -Key $Key){
        # to get the right etag, we must copy object back to itself
        $message = Copy-S3Object -SourceBucket $BucketName -SourceKey $Key -DestinationKey "$Key.copy"
        $message = Remove-S3Object -BucketName $BucketName -Key $Key -Force
        $message = Copy-S3Object -SourceBucket $BucketName -SourceKey "$Key.copy" -DestinationKey $Key
        $message = Remove-S3Object -BucketName $BucketName -Key "$Key.copy" -Force
    }
}
 
function Test-Zip {param ([String]$folder)
    $installFolder = "$scriptPath\$folder"
    if ((Test-Path -Path $installFolder\$magicSource ) -eq $false){
        Write-Output "Missing $folder folder containing $folder files."
        Write-Output "Run New-Item -ItemType directory -Path $installFolder\$magicSource"
        exit
    }
}
 
function New-Zip {param ([String]$folder)
    $installFolder = "$scriptPath\$folder"
    $timeZip = Measure-Command {
        & "$env:ProgramFiles\7-Zip\7z.exe" u "$installFolder.zip" "$installFolder\*.*" -r
    }
    Write-Output "Time to zip $folder was: $timeZip"
}
 
function Copy-Source {param ([String]$folder)
    Write-Output "Copying $folder to S3 if changed with Validation"
    Test-Zip -folder $folder
    New-Zip -folder $folder
    Copy-ZipS3 -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip"
    if ( (Test-S3CRC -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip") -ne 'ok' ){
        Write-Output 'MD5 did not match. Running Repair.'
        Repair-S3ETag -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip"
        Test-S3CRC -BucketName $bucket -Key "$folder.zip" -File "$scriptPath\$folder.zip" -Verbose
    }
}
 
Copy-Source -folder Install
Copy-Source -folder Config
Copy-Source -folder Package

References:
https://forums.aws.amazon.com/message.jspa?messageID=414257

score 0 · Answer 2 · edited Mar 12 '21 at 14:07

0

Write-S3Object -BucketName BUCKETNAME -File .\test.txt -Key another_key4 -Metadata @{ md5checksum = 'asdfasdfasdfsa'} -HeaderCollection @{ 'Content-Md5' = 'asdfasdfasdfsa'} -Verbose

Turn out you have to use the -HeaderCollection @{ 'Content-Md5' = 'asdfasdfasdfsa'} instead of -HeaderCollection @{ 'md5checksum' = 'asdfasdfasdfsa'}

edited Mar 12 '21 at 14:07

TheGameiswar

27,855
8
56
94

answered Mar 12 '21 at 13:25

Nick

920
1
7
21

How do you Write an S3 Object using AWS Tools in powershell and verify using an md5checksum

2 Answers2