How copy an existing object to same path Amazon S3

Question

Hi I want to copy an existing object to same path in AWS S3 and I am getting following exception

This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes

I am using Apache camel S3, how can i resolve this. After searching, i found there is a request header which we can use to replace the existing file but it is not working


// multiple other attempts also present, I am not sure which header will work

exchange.`in`.headers[AWS2S3Constants.METADATA] = mutableMapOf(
            "x-amz-metadata-directive" to "REPLACE",
            "x-amz-meta-directive" to "REPLACE",
            "metadata-directive" to "REPLACE",
            "MetadataDirective" to "REPLACE"
        )

I have logged in the request.

Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=, port=443, encodedPath=, headers=[amz-sdk-invocation-id, User-Agent, x-amz-copy-source, x-amz-meta-directive, x-amz-meta-metadata-directive, x-amz-meta-MetadataDirective, x-amz-meta-x-amz-metadata-directive], queryParameters=[])

But it is not working. how can i copy an existing object to same path without getting this error.

Why do you want to copy it to the same path? What does this achieve? — tgdavies, May 22 '22 at 07:19
I have use case, in which i need to skip processing file on weekend. But my S3 camel consumer, auto delete files after consumption. So on weekend i am polling and copy file to same path. If i turn of auto deletion, then i need to create an idempotent consumer (which is overkill in my scenario) — Zakir saifi, May 22 '22 at 07:25
hmm, yeah other option is use custom scheduler. do you have any idea how to configure in Apache camel? — Zakir saifi, May 22 '22 at 07:31
I have never used Camel, but you could start at https://camel.apache.org/components/3.16.x/cron-component.html — tgdavies, May 22 '22 at 07:45
Are you actually changing anything to do with the copied object? You need to change *something* e.g. file name? Please add the request & current code to the question. — Ermiya Eskandary, May 22 '22 at 08:28
works after updating filename with timestamp. but i have notice with aws-cli i was able to copy file same path aws s3 cp --metadata-directive REPLACE any idea @ErmiyaEskandary — Zakir saifi, May 22 '22 at 09:37
in aws doc it clearly say we can do that https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html — Zakir saifi, May 22 '22 at 10:47

score 0 · Answer 1 · answered May 30 '22 at 07:07

0

I end up changing the filename by suffixing it with a timestamp. Now I am no longer getting the error.

But I think, there should be some way to copy existing objects via API, since I am able to do same via aws-cli

answered May 30 '22 at 07:07

Zakir saifi

406
4
23

score 0 · Answer 2 · answered Jun 15 '22 at 12:48

Did you look at the copyObject operation in the Apache Camel AWS2 S3 component? https://camel.apache.org/components/3.17.x/aws2-s3-component.html#_s3_producer_operation_examples

There is an example related to headers needed to make the Copy Object operation works.

score 0 · Answer 3 · answered Oct 20 '22 at 11:26

$sourceBucket = '*** Your Source Bucket Name ***';
$sourceKeyname = '*** Your Source Object Key ***';
$targetBucket = '*** Your Target Bucket Name ***';

$s3 = new S3Client([
    'version' => 'latest',
    'region'  => 'us-east-1'
]);

// Copy an object.
$s3->copyObject([
    'Bucket'     => $targetBucket,
    'Key'        => "{$sourceKeyname}-copy",
    'CopySource' => "{$sourceBucket}/{$sourceKeyname}",
]);

How copy an existing object to same path Amazon S3

3 Answers3