3

I need to use Amazon S3 to serve my static and media files to my Django project.

However, I am facing a lot of issues with that. First, my code:

s3utils.py

from storages.backends.s3boto import S3BotoStorage

class FixedS3BotoStorage(S3BotoStorage):
def url(self, name):
    url = super(FixedS3BotoStorage, self).url(name)
    if name.endswith('/') and not url.endswith('/'):
        url += '/'
    return url

StaticS3BotoStorage = lambda: FixedS3BotoStorage(location='static')
MediaS3BotoStorage = lambda: FixedS3BotoStorage(location='media')

In settings.py

DEFAULT_FILE_STORAGE = 'SpareGuru.s3utils.MediaS3BotoStorage'
STATICFILES_STORAGE = 'SpareGuru.s3utils.StaticS3BotoStorage'

AWS_HOST = "s3-ap-southeast-1.amazonaws.com"
AWS_ACCESS_KEY_ID = 'xx'
AWS_SECRET_ACCESS_KEY = 'yy'
AWS_STORAGE_BUCKET_NAME = 'zz'

S3_URL = 'http://%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
MEDIA_DIRECTORY = '/media/'
STATIC_DIRECTORY = '/static/'

STATIC_URL = "/static/"

MEDIA_URL = "/media/"

STATIC_ROOT = S3_URL + STATIC_DIRECTORY
COMPRESS_ROOT = STATIC_ROOT
MEDIA_ROOT = S3_URL + MEDIA_DIRECTORY

Here are the issues I face:

  1. When running ./manage.py collectstatic, it starts to upload the files to S3 and after a couple of files, I get Broken Pipe error.

  2. When trying to run the webpage, I get the error: 'https://zz.s3.amazonaws.com/static/css/bootstrap.min.css?Signature=sign&Expires=1438359937&AWSAccessKeyId=xx' isn't accessible via COMPRESS_URL ('/static/') and can't be compressed.

No idea what's going on here.

Edit: My previous bucket policy allowed for Read only access. So maybe that could have been the reason that my compressor was unable to create files on S3. I have updated the policy but still doesn't work. Updated policy is:

{
"Statement": [
    {
        "Sid": "PublicReadForGetBucketObjects",
        "Effect": "Allow",
        "Principal": {
            "AWS": "*"
        },
        "Action": [
            "s3:GetObject"
        ],
        "Resource": [
            "arn:aws:s3:::zz/*"
        ]
    },
    {
        "Action": "s3:*",
        "Effect": "Allow",
        "Resource": [
            "arn:aws:s3:::zz",
            "arn:aws:s3:::zz/*"
        ],
        "Principal": {
            "AWS": [
                "my-arn:/goes=here"
            ]
        }
    }
]
}

And the CORS Configuration for my bucket is:

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>
Gaurav Wadhwani
  • 1,352
  • 2
  • 15
  • 32

2 Answers2

1

I have this running on my own Django project. I went and checked out the differences. One thing you might try is adding:

AWS_QUERYSTRING_AUTH = False

to your settings.py. That could help with at least your issue #2.

Here's a full example from my settings.py. Note: I don't use S3 for static files, only media.

from boto.s3.connection import SubdomainCallingFormat
AWS_CALLING_FORMAT = SubdomainCallingFormat()
DEFAULT_FILE_STORAGE = 'project.s3utils.MediaRootS3BotoStorage'
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID', None)
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_KEY', None)
AWS_SECRET_KEY = os.getenv('AWS_SECRET_KEY')
AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
AWS_QUERYSTRING_AUTH = False
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, '../',  'static')
MEDIA_ROOT = os.path.join(BASE_DIR, '../..',  'media')
MEDIA_URL = 'http://{!s}.s3.amazonaws.com/media/'.format(AWS_STORAGE_BUCKET_NAME)
Brice
  • 29
  • 4
  • Now I get: `'https://zz.s3.amazonaws.com/static/css/bootstrap.min.css' isn't accessible via COMPRESS_URL ('/static/') and can't be compressed`. The difference being I don't see the signature or access key ID anymore. But point being: it still doesn't work :( – Gaurav Wadhwani Jul 31 '15 at 18:33
  • Maybe it's an issue with django-compressor? What if you set: COMPRESS=False – Brice Jul 31 '15 at 18:53
  • Disabling Compressor using `COMPRESS_ENABLED = False` makes the site work. But the same code with compress works on local machine and not with S3. – Gaurav Wadhwani Jul 31 '15 at 19:01
  • Try adding: AWS_IS_GZIPPED = True – Brice Jul 31 '15 at 21:21
1

I believe your static and media urls/roots are incorrect. They should follow the format below:

MEDIA_URL = S3_URL + MEDIA_DIRECTORY
STATIC_URL = S3_URL + STATIC_DIRECTORY
MEDIA_ROOT = '/home/host/app/static_root/media'
STATIC_ROOT = '/home/host/app/static_root/static'

Note: Change the MEDIA_ROOT and STATIC_ROOT to how your project is configured.

EDIT:

Sorry, didn't see you were using compressor. Try doing the following:

COMPRESS_URL = S3_URL + STATIC_DIRECTORY
STATIC_URL = COMPRESS_URL
MEDIA_URL = S3_URL + MEDIA_DIRECTORY
MEDIA_ROOT = '/home/host/app/static_root/media'
STATIC_ROOT = '/home/host/app/static_root/static'
jape
  • 2,861
  • 2
  • 26
  • 58
  • Tried setting the `MEDIA_ROOT` and `STATIC_ROOT` as you described, but I still get `Uncompressable FileError`. – Gaurav Wadhwani Aug 01 '15 at 03:04
  • @GauravWadhwani Take a look at my edit. See if that works. – jape Aug 01 '15 at 03:20
  • Hi @jape , I still get `'https://zz.s3.amazonaws.com/static/css/bootstrap.min.css?Signature=sign%3D&Expires=1438403312&AWSAccessKeyId=xx' isn't accessible via COMPRESS_URL ('http://zz.s3.amazonaws.com/static/') and can't be compressed` – Gaurav Wadhwani Aug 01 '15 at 03:29
  • I've updated my bucket policy (and the question) to allow creation of files by the IAM user who's credentials are in `settings.py`. But it still doesn't work. – Gaurav Wadhwani Aug 01 '15 at 03:32
  • @GauravWadhwani On the bucket policy, change `PublicReadForGetBucketObjects` to `AllowPublicRead`. I would also add a new Permission for `Authenticated Users` (same place as bucket policy), and make all of the privileges allowed. If that doesn't work, what does your cors configuration look like? – jape Aug 01 '15 at 03:38
  • I have updated the question to show my `CORS Configuration`. And there are already 2 policies in that single bucket policy - one is to allow public read and the second grants the `principal` all privileges. – Gaurav Wadhwani Aug 01 '15 at 03:50