16

I have a large S3 bucket with a nested "folder" structure containing (among other things) static .json and .md files. Theses files are being served by S3 as text/plain rather than the correct application/json and text/markdown.

I have updated the bucket defaults so that new uploads will have the correct content type.

What is the best way to walk the "tree" and update the content type for files matching a certain extension?

Gabriel Bauman
  • 313
  • 1
  • 2
  • 10

4 Answers4

32

Here is an example how to do this with the aws cli tool. The cp tool allows the use of a recursive options, which I don't think the s3api tool can do. In this case, I'm fixing a bunch of SVGs. Remove the --dryrun options when you are ready to unleash it.

aws s3 cp \
      --exclude "*" \
      --include "*.svg" \
      --content-type="image/svg+xml"  \
      --metadata-directive="REPLACE" \
      --recursive \
      --dryrun \
       s3://mybucket/static/ \
       s3://mybucket/static/
t1m0
  • 436
  • 4
  • 4
  • I understand the underlying concept of the command. But will this delete the versions of the file since it is a replace directive? My bucket is configured with versioning and each file has some versions. Replacing it will delete all versions and make a new file or will it take into account all the versions during the replace? – Jose Luis Jun 05 '18 at 08:17
  • That’s a good question. Definitely worth a test in a throwaway bucket. My hope is that the request to replace just metadata wouldn’t blow away versions but... – t1m0 Jun 05 '18 at 12:37
  • I did a test, and turns out that versions stay, however the metadata modified version becomes the latest version, the older versions have the previous metadata. I am trying to find a way to recursively modify the metadata of all versions now, that is without making a complicated script that will list every version and then modify its metadata... – Jose Luis Jun 05 '18 at 13:35
  • 2
    This command resets acl. If you need non-default value, add `--acl=...` – Vasiliy Zverev Oct 21 '18 at 00:15
4

After some digging, I found that the s3cmd tool can do this. For example, to set JSON files to application/json:

s3cmd --recursive modify --acl-public \
       --add-header='content-type':'application/json' \
       --exclude '' --include '.json' \
       s3://bucket/
Gabriel Bauman
  • 313
  • 1
  • 2
  • 10
3

Unless you want to get your hands dirty using boto/python or another AWS API, I suspect your best bet would be to use $ aws s3api copy-object with the --content-type flag to copy an object to itself, setting the new content type.

Here's the documentation for s3api.

It goes without saying that you should test this first, before running it recursively on your whole bucket.

EEAA
  • 109,363
  • 18
  • 175
  • 245
3

As mentioned earlier, you can use aws tool by Amazon and use s3api to copy-object onto itself, and use metadata-directive=REPLACE to change the content-type.

I am putting this here, because sometimes you would want to iterate on filenames that are stored in database, and this is how you can do it through cli. 

aws s3api copy-object \
          --content-type="application/vnd.android.package-archive" \
          --metadata-directive="REPLACE" \
          --copy-source "MYBUCKET/FILE.apk" \
          --bucket "MYBUCKET" \
          --key "FILE.apk" \
          --acl public-read
gilm
  • 143
  • 6