How to cp file only if it does not exist, throw error otherwise?

Question

aws s3 cp "dist/myfile" "s3://my-bucket/production/myfile"

It always copies myfile to s3 - I would like to copy file ONLY if it does no exist, throw error otherwise. How I can do it? Or at least how I can use awscli to check if file exists?

Answer 1

You could test for the existence of a file by listing the file, and seeing whether it returns something. For example:

aws s3 ls s3://bucket/file.txt | wc -l

This would return a zero (no lines) if the file does not exist.

---

If you only want to copy a file if it does not exist, try the sync command, e.g.:

aws s3 sync . s3://bucket/ --exclude '*' --include 'file.txt'

This will synchronize the local file with the remote object, only copying it if it does not exist or if the local file is different to the remote object.

Answer 2

So, turns out that "aws s3 sync" doesn't do files, only directories. If you give it a file, you get...interesting...behavior, since it treats anything you give it like a directory and throws a slash on it. At least aws-cli/1.6.7 Python/2.7.5 Darwin/13.4.0 does.

    %% date > test.txt
    %% aws s3 sync test.txt s3://bucket/test.txt
    warning: Skipping file /Users/draistrick/aws/test.txt/. File does not exist.

So, if you -really- only want to sync a file (only upload if exists, and if checksum matches) you can do it:

    file="test.txt"
    aws s3 sync --exclude '*' --include "$file" "$(dirname $file)" "s3://bucket/"

Note the exclude/include order - if you reverse that, it won't include anything. And your source and include path need to have sanity around their matching, so maybe a $(basename $file) is in order for --include if you're using full paths... aws --debug s3 sync is your friend here to see how the includes evaluate.

And don't forget the target is a directory key, not a file key.

Here's a working example:

  %% file="test.txt"
  %% date >> $file
  %% aws s3 sync --exclude '*' --include "$file" "$(dirname $file)" "s3://bucket/"
  upload: ./test.txt to s3://bucket/test.txt/test.txt
  %% aws s3 sync --exclude '*' --include "$file" "$(dirname $file)" "s3://bucket/"
  %% date >> $file
  %% aws s3 sync --exclude '*' --include "$file" "$(dirname $file)" "s3://bucket/"
  upload: ./test.txt to s3://bucket/test.txt/test.txt

(now, if only there were a way to ask aws s3 to -just- validate the checksum, since it seems to always do multipart style checksums.. oh, maybe some --dryrun and some output scraping and sync..)

Answer 3

You can do this by listing and copying if and only if the list succeeds.

aws s3 ls "s3://my-bucket/production/myfile" || aws s3 cp "dist/myfile" "s3://my-bucket/production/myfile"

Edit: replaced && to || to have the desired effect of if list fails do copy

Answer 4

You can also check the existence of a file by aws s3api head-object subcommand. An advantage of this over aws s3 ls is that it just requires s3:GetObject permission instead of s3:ListBucket.

$ aws s3api head-object --bucket ${BUCKET} --key ${EXISTENT_KEY}
{
    "AcceptRanges": "bytes",
    "LastModified": "Wed, 1 Jan 2020 00:00:00 GMT",
    "ContentLength": 10,
    "ETag": "\"...\"",
    "VersionId": "...",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}
$ echo $?
0

$ aws s3api head-object --bucket ${BUCKET} --key ${NON_EXISTENT_KEY}

An error occurred (403) when calling the HeadObject operation: Forbidden
$ echo $?
255

Note that the HTTP status code for the non-existent object depends on whether you have the s3:ListObject permission. See the API document for more details:

• If you have the s3:ListBucket permission on the bucket, Amazon S3 returns an HTTP status code 404 ("no such key") error.
• If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 ("access denied") error.

Answer 5

AWS HACK

You can run the following command to raise ERROR if the file already exists

• Run aws s3 sync command to sync the file to s3, it will return the copied path if the file doesn't exist or it will give blank output if it exits
• Run wc -c command to check the character count and raise an error if the output is zero

com=$(aws s3 sync dist/ s3://my-bucket/production/ | wc -c);if [[ $com -ne 0 ]]; then exit 1; else exit 0; fi;

OR

#!/usr/bin/env bash
com=$(aws s3 sync dist s3://my-bucket/production/ | wc -c)
echo "hello $com"
if [[ $com -ne 0 ]]; then
echo "File already exists"
exit 1
else
echo "success"
exit 0
fi

Answer 6

I voted up aviggiano. Using his example above, I was able to get this to work in my windows .bat file. If the S3 path exists it will throw an error and end the batch job. If the file does not exist it will continue on to perform the copy function. Hope this helps some one.

:Step1

aws s3 ls s3://00000000000-fake-bucket/my/s3/path/inbound/test.txt && ECHO Could not copy to S3 bucket becasue S3 Object already exists, ending script. && GOTO :Failure

ECHO No file found in bucket, begin upload.

aws s3 cp Z:\MY\LOCAL\PATH\test.txt s3://00000000000-fake-bucket/my/s3/path/inbound/test.txt --exclude "*" --include "*.txt"


:Step2

ECHO YOU MADE IT, LET'S CELEBRATE

IF %ERRORLEVEL% == 0 GOTO :Success
GOTO :Failure

:Success
echo Job Endedsuccess
GOTO :ExitScript

:Failure
echo BC_Script_Execution_Complete Failure
GOTO :ExitScript

:ExitScript

Answer 7

I am running AWS on windows. and this is my simple script.

rem clean work files:

if exist  SomeFileGroup_remote.txt del /q SomeFileGroup_remote.txt
if exist  SomeFileGroup_remote-fileOnly.txt del /q SomeFileGroup_remote-fileOnly.txt
if exist  SomeFileGroup_Local-fileOnly.txt del /q SomeFileGroup_Local-fileOnly.txt
if exist  SomeFileGroup_remote-Download-fileOnly.txt del /q SomeFileGroup_remote-Download-fileOnly.txt

Rem prep:

call F:\Utilities\BIN\mhedate.cmd
aws s3 ls s3://awsbucket//someuser@domain.com/BulkRecDocImg/folder/folder2/ --recursive >>SomeFileGroup_remote.txt
for /F "tokens=1,2,3,4* delims= " %%i in (SomeFileGroup_remote.txt) do @echo %%~nxl >>SomeFileGroup_remote-fileOnly.txt
dir /b temp\*.* >>SomeFileGroup_Local-fileOnly.txt
findstr  /v /I /l /G:"SomeFileGroup_Local-fileOnly.txt" SomeFileGroup_remote-fileOnly.txt >>SomeFileGroup_remote-Download-fileOnly.txt

Rem Download:

for /F "tokens=1* delims= " %%i in (SomeFileGroup_remote-Download-fileOnly.txt) do (aws s3 cp s3://awsbucket//someuser@domain.com/BulkRecDocImg/folder/folder2/%%~nxi "temp" >>"SomeFileGroup_Download_%DATE.YEAR%%DATE.MONTH%%DATE.DAY%.log")

Answer 8

I Added Date to the path in-order to not override the file:

 aws cp videos/video_name.mp4 s3://BUCKET_NAME/$(date +%D-%H:%M:%S)

So that way I will have history and the existing file won't be overriddend.