Django AWS S3 using Boto with Compressor fails to compress UncompressableFileError

Question

Following this guide, and these [1] [2] posts, I have tried to setup static storage on AWS S3 using django-storages Boto.

When executing collectstatic, the command succesfully collects at the STATIC_ROOT. However, the files are not uploaded to S3 compressed, and the server can't serve them. A 500 error is returned. Looking at the logs:

Error Message:

UncompressableFileError: 'https://<myapp>.s3.amazonaws.com/static/oscar/css/styles.css' could not be found in the COMPRESS_ROOT '/var/www/<myappname>/static' or with staticfiles.

EDIT:

I also changed STATIC_URL to http://%s/ % AWS_S3_CUSTOM_DOMAIN, and I get the same error, except that it is still searching at https, but the COMPRESS_URL is http.

UncompressableFileError: 'https://<myappname>.s3.amazonaws.com/static/oscar/css/styles.css' isn't accessible via COMPRESS_URL ('http://<myappname>.s3.amazonaws.com/') and can't be compressed

Is this an incompatibility with compressor and Boto?

relevant Code:

# settings/prod.py

AWS_ACCESS_KEY_ID = <Key_ID>
AWS_SECRET_ACCESS_KEY = <Secret_Key>
AWS_STORAGE_BUCKET_NAME = "<my_bucket_name>"
AWS_S3_CUSTOM_DOMAIN = "%s.s3.amazonaws.com" % AWS_STORAGE_BUCKET_NAME
STATIC_URL = "https://%s/" % AWS_S3_CUSTOM_DOMAIN
AWS_LOCATION = 'static'

DEFAULT_FILE_STORAGE = "storages.backends.s3boto.S3BotoStorage"
STATICFILES_STORAGE = "myapp.storage.s3utils.CachedS3BotoStorage"

COMPRESS_STORAGE = "myapp.storage.s3utils.CachedS3BotoStorage"
AWS_IS_GZIPPED = True
COMPRESS_URL = STATIC_URL
STATIC_ROOT = "/var/www/<myappname>/static/"
COMPRESS_ROOT = STATIC_ROOT

storage/s3utils.py from this documentation

from django.core.files.storage import get_storage_class
from storages.backends.s3boto import S3BotoStorage

class CachedS3BotoStorage(S3BotoStorage):
    """
    S3 storage backend that saves the files locally, too.
    """
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class(
            "compressor.storage.CompressorFileStorage")()

    def save(self, name, content):
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name

Can you run the collectstatic command with --traceback that may give a bit more detail as to what's going wrong. I remember having several problems with compressor though. If you ever plan on using cloudfront with this just be aware that the CORS settings are very very very painful to get right in AWS. — Paul, Feb 18 '16 at 21:00

score 5 · Accepted Answer · answered Feb 19 '16 at 03:35

Solved with these settings:

AWS_ACCESS_KEY_ID = '<KEY_ID>'
AWS_SECRET_ACCESS_KEY = '<SECRET_KEY>'
AWS_STORAGE_BUCKET_NAME = "<app_name>"
AWS_S3_CUSTOM_DOMAIN = "s3.amazonaws.com/%s" % AWS_STORAGE_BUCKET_NAME

MEDIA_URL = "https://%s/media/" % AWS_S3_CUSTOM_DOMAIN
STATIC_URL = "https://%s/static/" % AWS_S3_CUSTOM_DOMAIN
COMPRESS_URL = STATIC_URL

DEFAULT_FILE_STORAGE = '<app_name>.storage.s3utils.MediaS3BotoStorage'
STATICFILES_STORAGE = '<app_name>.storage.s3utils.CachedS3BotoStorage'
COMPRESS_STORAGE = '<app_name>.storage.s3utils.CachedS3BotoStorage'


MEDIA_ROOT = '<app_name>/media/'
STATIC_ROOT = '<app_name>/static/'
COMPRESS_ROOT = STATIC_ROOT


COMPRESS_ENABLED = True
COMPRESS_CSS_FILTERS = ['compressor.filters.css_default.CssAbsoluteFilter',
                        'compressor.filters.cssmin.CSSMinFilter'
                       ]
COMPRESS_PARSER = 'compressor.parser.HtmlParser'

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'compressor.finders.CompressorFinder'
)

and my s3utils.py

class CachedS3BotoStorage(S3BotoStorage):
    """
    S3 storage backend that saves files locally too. 
    """
    location = 'static'
    def __init__(self, *args, **kwargs):
        super(CachedS3BotoStorage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class(
            "compressor.storage.CompressorFileStorage")()

    def save(self, name, content):
        name = super(CachedS3BotoStorage, self).save(name, content)
        self.local_storage._save(name, content)
        return name



class MediaS3BotoStorage(S3BotoStorage):
    """ S3 storage backend that saves to the 'media' subdirectory"""
    location = 'media'

Great answer, I have little different setup and I was pulling my hair off and what helped me I added https: - hardcoded instead of // to my code and it worked, thank you the question is here if it helps to someone. http://stackoverflow.com/questions/40825990/django-compressor-throws-uncompressablefileerror-with-django-storages-using-amaz/40832184#40832184 — Radek, Nov 27 '16 at 18:19

score 0 · Answer 2 · answered Feb 18 '16 at 23:24

Looks like someone had the same problem here: https://github.com/django-compressor/django-compressor/issues/368#issuecomment-182817810

Try this:

import copy

def save(self, name, content):
    content2 = copy.copy(content)
    name = super(CachedS3BotoStorage, self).save(name, content)
    self.local_storage._save(name, content2)
    return name

Note: I'm using django-storages S3BotoStorage & django-compressor together without issue. I think it's the gzipping that's causing problems.

Django AWS S3 using Boto with Compressor fails to compress UncompressableFileError

2 Answers2