Checking for successful S3 copy operation?

Question

I'm trying to implement a move operation using the Amazon S3 Java API.

The problem I am having is that the CopyObjectResult object returned by the AmazonS3Client.copyObject method doesn't seem to have a clear indicator about wiether the operation was successful or not.

Given that after this operation I am going to be deleting a file, I'd want to make sure that the operation was successful.

Any suggestions?

Do you have reason to believe that the method can fail without throwing an exception? — Aleksandr Dubinsky, Jul 20 '14 at 16:19

score 6 · Accepted Answer · answered Jul 11 '13 at 14:59

6

This is what I ended up doing,

def s3 = createS3Client(credentials)
def originalMeta = s3.getObjectMetadata(originalBucketName, key)
s3.copyObject(originalBucketName, key, newBucketName, newKey)
def newMeta = s3.getObjectMetadata(newBucketName, newKey)

// check that md5 matches to confirm this operation was successful
return originalMeta.contentMD5 == newMeta.contentMD5

Note that this is Groovy code, but it is extremely similar to how the Java code would work.

I don't like having to make two additional operations to check the metadata, so if there is anyway to do this more efficiently let me know.

answered Jul 11 '13 at 14:59

James McMahon

48,506
64
207
283

This is pretty similar to what I recently did with the PHP API. I'd capture the originals etag, copy, compare original etag to copied etag, if match, delete original. – Dan Jul 16 '13 at 21:36
Historical note, following up on Dan's comment: It appears that the system for generating etags changes when a multipart upload is used as opposed to a regular upload, so comparison by MD5 might be the better approach. – Shotgun Ninja Mar 25 '14 at 15:37
2

Since S3 doesn't have immediate consistency, wouldn't this method occasionally show that the object hasn't been copied even though it has? – Aleksandr Dubinsky Jul 20 '14 at 16:20

score 1 · Answer 2 · answered Jul 10 '13 at 22:01

1

I'm pretty sure you can just use CopyObjectResult object's getETag method to confirm that, after created, it has a valid ETag, as was made in the CopyObjectRequest setETag method.

getETag

public String getETag() Gets the ETag value for the new object that was created in the associated CopyObjectRequest. Returns: The ETag value for the new object. See Also: setETag(String) setETag

public void setETag(String etag) Sets the ETag value for the new object that was created from the associated copy object request. Parameters: etag - The ETag value for the new object. See Also: getETag()

Always trust the data.

Been a year since I did a similar function with the Amazon PhP SDK a couple years ago, but it should work.

answered Jul 10 '13 at 22:01

jdero

1,797
3
21
36

1

This is probably just me being unfamiliar with S3, but what explicitly would I check for on the etag string? – James McMahon Jul 10 '13 at 22:02
1

If you set it when you made your CopyObjectRequest, you should be able to compare the two for equality. Hope that helps. – jdero Jul 10 '13 at 22:04
Ultimately, if you're unsure, just test it. Use an file you don't care about. – jdero Jul 10 '13 at 22:06
1

I feel like I am missing some larger context, what is an ETag? – James McMahon Jul 10 '13 at 23:31
1

They explain it well in this thread: http://stackoverflow.com/questions/6591047/etag-definition-changed-in-amazon-s3 – jdero Jul 11 '13 at 07:35

score 0 · Answer 3 · answered Jan 07 '21 at 01:51

0

AWS documentation says

The source and destination ETag is identical for a successfully copied

answered Jan 07 '21 at 01:51

Anuruddha

1,367
6
19
38

Checking for successful S3 copy operation?

3 Answers3

Linked