Django-storages + boto + S3 minimal credentials for collectstatic

Question

I am using django-storages with the storages.backends.s3boto.S3BotoStorage and have come across a strange 403 error.

My original IAM policy was quite conservative, and only included get, put, and delete for objects.

--> This threw a 403 error

I then gave all permissions except the permissions to delete and create buckets

--> To my surprise, this also threw the 403 error

I finally gave full permissions, which I would like to avoid, and I am no longer getting the 403 error.

I have already tried providing access to both the bucket root / and /* as per this answer

My objective is to give only the permissions necessary.

score 3 · Answer 1 · edited May 23 '17 at 10:28

Here's the bucket policy for uploading to a specific bucket, in this case static.

{
    "Version": "2008-10-17",
    "Id": "StaticAndMediaPermissions",
    "Statement": [
        {
            "Sid": "AllowAnybodyToGetBucketLocation",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::<bucket_name>"
        },
        {
            "Sid": "AllowCollectstaticUserToListStaticDirectory",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<collectstatic_user_arn_from_iam_user_summary>"
            },
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name>/static"
            ]
        },
        {
            "Sid": "AllowCollectstaticUserAccessToAllObjectsInStaticDirectory",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<collectstatic_user_arn_from_iam_user_summary>"
            },
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObjectAcl",
                "s3:ListMultipartUploadParts",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucket_name>/static/*"
        }
    ]
}

You will still need to add the IAM user policy from my other answer.

Make sure you have the AWS_LOCATION variable set to /static/ in your settings.py. If you don't have that set, this will not work.

Sources: * http://blogs.aws.amazon.com/security/post/Tx1P2T3LFXXCNB5/Writing-IAM-policies-Grant-access-to-user-specific-folders-in-an-Amazon-S3-bucke - Walked through how to set directory-specific permissions * https://stackoverflow.com/a/9649233/1999151 - Clued me into using AWS_LOCATION

Alternatively, if you want to have separate directories for static and media files, you will need to follow the directions here:

https://stackoverflow.com/a/10626241/1999151

and remove AWS_LOCATION from settings.py.

You will still need to add the IAM user policy from my other answer.

Then you will need to adjust the AWS bucket policy to the following:

{
    "Version": "2008-10-17",
    "Id": "StaticAndMediaPermissions",
    "Statement": [
        {
            "Sid": "AllowAnybodyToGetBucketLocation",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::<bucket_name>"
        },
        {
            "Sid": "AllowCollectstaticUserToListStaticAndMediaDirectories",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<collectstatic_user_arn_from_iam_user_summary>"
            },
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name>/media"
                "arn:aws:s3:::<bucket_name>/static"
            ]
        },
        {
            "Sid": "AllowCollectstaticUserAccessToAllObjectsInStaticAndMediaDirectories",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<collectstatic_user_arn_from_iam_user_summary>"
            },
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObjectAcl",
                "s3:ListMultipartUploadParts",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>/media/*",
                "arn:aws:s3:::<bucket_name>/static/*"
            ]
        }
    ]
}

score 1 · Answer 2 · edited May 23 '17 at 12:10

I was trying to do the same thing, and I think I finally got it working (with a lot of Googling and a lot of other StackOverflow answers). This isn't guaranteed to be the minimal required set of permissions, but it isn't a whole lot.

Here is the IAM policy for the user collectstatic is using:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1399990928000",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    }
  ]
}

That seems to be pretty straightforward.

From here, it gets more complicated. Things you need to change are marked within angle brackets. The bucket policy should be as follows:

{
    "Version": "2008-10-17",
    "Id": "Permissions",
    "Statement": [
        {
            "Sid": "AllowAnybodyToGetBucketLocation",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::<bucket_name>"
        },
        {
            "Sid": "AllowCollectstaticUserToListBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<collectstatic_user_arn_from_iam_user_summary>"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<bucket_name>"
        },
        {
            "Sid": "AllowCollectstaticUserAccessToAllObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<collectstatic_user_arn_from_iam_user_summary>"
            },
            "Action": [
                "s3:DeleteObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucket_name>/*"
        }
    ]
}

That gives the collectstatic's user at least the minimum required permissions.

Note that this assumes that you are dumping all of your objects directly into the root of the bucket and NOT into a directory like static/. I'm not entirely sure how to do that - whether I should specify the Resource as arn:aws:s3:::<bucket_name>/static/* or whether to add a Condition onto AllowCollectstaticUserAccessToAllObjects.

Sources that helped me:

http://blog.iambob.me/the-super-stupid-idiots-guide-to-getting-started-with-django-pipeline-and-s3/
- Helped me get started with S3 configuration
https://stackoverflow.com/a/10884964
- Need to give permissions to both the bucket itself and child objects
https://stackoverflow.com/a/10894488
- Figured out if it was a permissions issue on S3 or a django-storages configuration issue
https://stackoverflow.com/a/13658829
- ListAllBuckets has to be done in IAM user policy
http://paltman.com/2007/09/21/detailed-amazon-s3-logging/
- Once I setup logging, this post helped me decode the logs
https://stackoverflow.com/a/19425526
- Apparently you can specify globs of permissions, which allowed me to test s3:Delete*, s3:Put*, and s3:Get* separately.

Jonathan Richards · Answer 3 · 2017-09-19T14:23:30.387

I'm assuming this has changed in recent years, but I was just able to collect static with the following permissions:

{
    "Statement": [
        {
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<bucket_name>/*"
            ]
        }
    ]
}

The permission that was tripping me up was the PutObjectAcl. I'm also not sure if the multipart stuff is necessary, but I don't want to find out its necessary for large files or something like that down the line.

Edit: Apparently django-storages will delete files in some situations. I was able to get it to work with:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>"
            ]
        },
        {
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject",
                "s3:DeleteObjectTagging",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<bucket_name>/*"
            ]
        }
    ]
}

There may be a couple extra fields here, but I'm having trouble reproducing a delete operation to confirm.

score 0 · Answer 4 · answered Jul 07 '23 at 05:56

putting permission policy aside, My issue was that i was creating bucket with root user. even though i provide access to IAM user for the bucket but still was getting 403 forbidden then I create the bucket from IAM user by providing IAM user the console access. and it works out.

Django-storages + boto + S3 minimal credentials for collectstatic

4 Answers4

Linked