AWS S3 CLI CP file and add metadata

Question

Trying to copy a local file named test.txt to my s3 bucket and add metadata to the file.

But it always prints error:

argument --metadata-directive: Invalid choice, valid choices are: COPY | REPLACE

Is it possible to do this with the cp command, as I understand the docs it should be possible. AWS CLI CP DOCS

This is the commands I've tried:

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata x-amz-meta-cms-id:34533452

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata-directive COPY --metadata x-amz-meta-cms-id:34533452

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata-directive COPY --metadata '{"x-amz-meta-cms-id":"34533452"}'

aws s3 cp test.txt to s3://a-bucket/test.txt --metadata '{"x-amz-meta-cms-id":"34533452"}'

aws --version: aws-cli/1.9.7 Python/2.7.10 Darwin/16.1.0 botocore/1.3.7

OS: macOS Sierra version 10.12.1

Edit

Worth mentioning is that uploading a file without the --metadata flag works fine.

Hmm, I've checked the help for my version of cli with aws s3 cp help Turns out it does not list --metadata as an option, as the docs at the given link above does.

If runnig older version of aws cli

Use aws s3api put-object

How to upload a file to a bucket and add metadata:

aws s3api put-object --bucket a-bucket --key test.txt --body test.txt --metadata '{"x-amz-meta-cms-id":"34533452"}'

Docs: AWS S3API DOCS

--metadata "filename=foobar.text" worked for me on cli 1.16.40 — miracle_the_V, Nov 05 '18 at 17:01

Frederic Henri · Accepted Answer · 2016-11-16T09:00:27.673

48

Indeed the support for metadata option has been added since 1.9.10

aws s3 Added support for custom metadata in cp, mv, and sync.

so upgrading your aws cli to this version (or even better to latest) - and the metadata value needs to be a map so

aws s3 cp test.txt s3://a-bucket/test.txt --metadata '{"x-amz-meta-cms-id":"34533452"}'

edited Nov 16 '16 at 09:00

answered Nov 16 '16 at 08:41

Frederic Henri

51,761
10
113
139

Thanks mate! For some reason I read 1.9 as a higher version that 1.11 – Jonathan Andersson Nov 16 '16 at 08:57
For me the command the json had to be inside ' ' aws s3 cp test.txt s3://a-bucket/test.txt --metadata '{"x-amz-meta-cms-id":"34533452"}' – Jonathan Andersson Nov 16 '16 at 08:59
hum ... ok will edit, I did not retry honestly - thought it worked without the quote. will edit so it works as an answer – Frederic Henri Nov 16 '16 at 09:00
7

I'm not sure you need the `x-amz-meta-` in the key name. This seems to get added automatically so you will end up with `x-amz-meta-x-amz-meta-cms-id`. – Ade Miller Sep 19 '17 at 19:53
This works for me in linux but not in windows – Chris Maggiulli Jul 19 '21 at 18:14
The metadata can also be specified in the key/value notation: `--metadata KeyName1=string,KeyName2=string` – Erik Finnman Feb 16 '23 at 10:49

score 6 · Answer 2 · edited Aug 29 '19 at 13:50

6

Install s3cmd tools (free) and invoke like so:

s3cmd modify --add-header x-amz-meta-foo:bar s3://<bucket>/<object>

With x-amz-meta-foo:bar header you will get foo as key and bar as value of that key.

edited Aug 29 '19 at 13:50

Paul T. Rawkeen

3,994
3
35
51

answered May 24 '17 at 11:20

Srini Sydney

564
8
17

Don't know why this was voted down, it seems to work (without copying the object) `s3cmd modify --add-header x-amz-meta-foo:bar s3://ubookrcd/supplier-data/3101/campaigns/45367//original` `aws s3api head-object --bucket ubookrcd --key supplier-data/3101/campaigns/45367/43//original { "AcceptRanges": "bytes", "ContentType": "image/jpeg", "LastModified": "Tue, 05 Jun 2018 16:20:46 GMT", "ContentLength": 5698870, "ETag": "\"7a36ba8566eaf11abede38c0e3f01cc2\"", "Metadata": { "qqfilename": "Image_23.jpg", "foo": "bar" } }` – Shanness Jun 05 '18 at 16:28

score 1 · Answer 3 · answered Aug 29 '22 at 18:29

1

There are special flags to set Content-Type and Content-Encoding

aws s3 cp test.gz. s3://a-bucket/test.gz --content-type application/octet-stream --content-encoding gzip

answered Aug 29 '22 at 18:29

almeynman

7,088
3
23
37

score 0 · Answer 4 · edited Apr 13 '17 at 12:13

There is bug with metadata directive "COPY" option.

aws s3api copy-object --bucket testkartik --copy-source testkartik/costs.csv --key costs.csv --metadata-directive "COPY" --metadata "SomeKey=SomeValue"

Below are the three steps to understand cli command with JQ workaround.

Install JQ library to deal with json metadata using command line.
Read the existing metadata.

aws s3api head-object --bucket <bucket> --key <key> | jq '.Metadata' | jq --compact-output '. +{"new":"metadata", "another" : "metadata"}'
Add new metadata.

aws s3api copy-object --bucket <bucket-name> --copy-source <bucket/key> --key <key> --metadata-directive "REPLACE" --metadata $(READ-THE-EXISTING-From-Step-2)

Complete command in one go.

aws s3api copy-object --bucket <bucket-name> --copy-source <bucket/key> --key <key> --metadata-directive "REPLACE" --metadata $(aws s3api head-object --bucket <bucket> --key <key> | jq '.Metadata' | jq --compact-output '. +{"new":"metadata", "another" : "metadata"}')

Upvoted. This method works if for some reason you cannot use the new version of cp. — jmathew, Jan 22 '22 at 17:54

AWS S3 CLI CP file and add metadata

4 Answers4

Linked