How to set-up a Django project with django-storages and Amazon S3, but with different folders for static files and media files?

Question

I'm configuring a Django project that were using the server filesystem for storing the apps static files (STATIC_ROOT) and user uploaded files (MEDIA_ROOT).

I need now to host all that content on Amazon's S3, so I have created a bucket for this. Using django-storages with the boto storage backend, I managed to upload collected statics to the S3 bucket:

MEDIA_ROOT = '/media/'
STATIC_ROOT = '/static/'

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'KEY_ID...'
AWS_SECRET_ACCESS_KEY = 'ACCESS_KEY...'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

Then, I got a problem: the MEDIA_ROOT and STATIC_ROOT are not used within the bucket, so the bucket root contains both the static files and user uploaded paths.

So then I could set:

S3_URL = 'http://s3.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME
STATIC_URL = S3_URL + STATIC_ROOT
MEDIA_URL = 'S3_URL + MEDIA_ROOT

And use those settings in the templates, but there is no distinction of static/media files when storing in S3 with django-storages.

How this can be done?

Thanks!

Because there's only one setting to specify the bucket's name (`AWS_STORAGE_BUCKET_NAME`), and that's the one used when an instance of the class specified in `STATICFILES_STORAGE` is instantiated. — Armando Pérez Marqués, May 16 '12 at 20:12

bradenm · Accepted Answer · 2013-10-23T03:19:27.867

129

I think the following should work, and be simpler than Mandx's method, although it's very similar:

Create a s3utils.py file:

from storages.backends.s3boto import S3BotoStorage

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage  = lambda: S3BotoStorage(location='media')

Then in your settings.py:

DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3BotoStorage'

A different but related example (that I've actually tested) can be seen in the two example_ files here.

edited Oct 23 '13 at 03:19

answered May 30 '12 at 23:32

bradenm

2,150
1
17
10

1

Definitely simpler and better than my version. Although I have not tested this, I also think this will work. Thanks! I'm also checking your [django-s3storage](https://github.com/mstarinc/django-s3storage/) repo, seems a very lightweight solution if the project uses S3 exclusively. – Armando Pérez Marqués May 31 '12 at 03:37
1

And, if you are more into packaging, check out [django-s3-folder-storage](http://pypi.python.org/pypi/django-s3-folder-storage/). I just found it, can't tell if it's this very same solution but prepackaged. – Armando Pérez Marqués Jul 04 '12 at 01:36
Why is `SimpleLazyObject` imported? – glarrain Mar 05 '13 at 19:43
Sorry, `SimpleLazyObject` was there because I copy-pasted this from code that I use. – bradenm Mar 06 '13 at 02:43
4

This doesn't work form me, media files are uploaded to the / of the s3 bucket. Seems location setting is not being respected. django-storages==1.1.6, django-extensions==1.1.1, django=1.4 – Nathan Keller Mar 21 '13 at 20:40
3

For it me it made more sense to have separate buckets and I don't like having config outside of my settings module so my solution ended up looking like this https://gist.github.com/antonagestam/6075199 – antonagestam Jul 24 '13 at 22:25
With this approach, do I have to leave out the `aws:elasticbeanstalk:container:python:staticfiles` config option from the .config file? In the documentation it says it's a "A mapping of the URL to a local directory." But I do not want it to be in the local directory – barrigaj Dec 20 '13 at 01:14
1

This solutions doesn't work, from what I can tell. This should be the approach: https://gist.github.com/defrex/82680e858281d3d3e6e4 – defrex Jul 14 '14 at 19:02
1

I had to subclass the `S3BotoStorage` class, as the instantiation in the lambda function caused an exception. `class MediaS3BotoStorage(S3BotoStorage):` `upload_to='media'` for example – errkk Jan 20 '15 at 14:10
Flake8 doesn't like the use of lambda above: E731 do not assign a lambda expression, use a def – Ronnie Beltran Feb 17 '17 at 07:41

score 8 · Answer 2 · answered May 16 '12 at 20:31

I'm currently using this code in a separated s3utils module:

from django.core.exceptions import SuspiciousOperation
from django.utils.encoding import force_unicode

from storages.backends.s3boto import S3BotoStorage


def safe_join(base, *paths):
    """
    A version of django.utils._os.safe_join for S3 paths.

    Joins one or more path components to the base path component intelligently.
    Returns a normalized version of the final path.

    The final path must be located inside of the base path component (otherwise
    a ValueError is raised).

    Paths outside the base path indicate a possible security sensitive operation.
    """
    from urlparse import urljoin
    base_path = force_unicode(base)
    paths = map(lambda p: force_unicode(p), paths)
    final_path = urljoin(base_path + ("/" if not base_path.endswith("/") else ""), *paths)
    # Ensure final_path starts with base_path and that the next character after
    # the final path is '/' (or nothing, in which case final_path must be
    # equal to base_path).
    base_path_len = len(base_path) - 1
    if not final_path.startswith(base_path) \
       or final_path[base_path_len:base_path_len + 1] not in ('', '/'):
        raise ValueError('the joined path is located outside of the base path'
                         ' component')
    return final_path


class StaticRootS3BotoStorage(S3BotoStorage):
    def __init__(self, *args, **kwargs):
        super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)
        self.location = kwargs.get('location', '')
        self.location = 'static/' + self.location.lstrip('/')

    def _normalize_name(self, name):
        try:
            return safe_join(self.location, name).lstrip('/')
        except ValueError:
            raise SuspiciousOperation("Attempted access to '%s' denied." % name)


class MediaRootS3BotoStorage(S3BotoStorage):
    def __init__(self, *args, **kwargs):
        super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)
        self.location = kwargs.get('location', '')
        self.location = 'media/' + self.location.lstrip('/')

    def _normalize_name(self, name):
        try:
            return safe_join(self.location, name).lstrip('/')
        except ValueError:
            raise SuspiciousOperation("Attempted access to '%s' denied." % name)

Then, in my settings module:

DEFAULT_FILE_STORAGE = 'myproyect.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myproyect.s3utils.StaticRootS3BotoStorage'

I got to redefine the _normalize_name() private method to use a "fixed" version of the safe_join() function, since the original code is giving me SuspiciousOperation exceptions for legal paths.

I'm posting this for consideration, if anyone can give a better answer or improve this one, it will be very welcome.

score 7 · Answer 3 · answered Mar 10 '15 at 18:06

File: PROJECT_NAME/custom_storages.py

from django.conf import settings
from storages.backends.s3boto import S3BotoStorage

class StaticStorage(S3BotoStorage):
    location = settings.STATICFILES_LOCATION

class MediaStorage(S3BotoStorage):
    location = settings.MEDIAFILES_LOCATION

File: PROJECT_NAME/settings.py

STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'

if not DEBUG:
    STATICFILES_STORAGE = 'PROJECT_NAME.custom_storages.StaticStorage'
    DEFAULT_FILE_STORAGE = 'PROJECT_NAME.custom_storages.MediaStorage'
    AWS_ACCESS_KEY_ID = 'KEY_XXXXXXX'
    AWS_SECRET_ACCESS_KEY = 'SECRET_XXXXXXXXX'
    AWS_STORAGE_BUCKET_NAME = 'BUCKET_NAME'
    AWS_HEADERS = {'Cache-Control': 'max-age=86400',}
    AWS_QUERYSTRING_AUTH = False

And run: python manage.py collectstatic

If you happen to name this file `storages.py` instead of `custom_storages.py` You'll want to use `from __future__ import absolute_import` — Aaron McMillin, May 10 '17 at 20:57

score 2 · Answer 4 · answered Oct 01 '14 at 16:15

I think the answer is pretty simple and done by default. This is working for me on AWS Elastic Beanstalk with Django 1.6.5 and Boto 2.28.0:

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)

TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.Loader',
    'django.template.loaders.app_directories.Loader',
)

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_KEY']

The AWS keys are passed in from the container config file and I have no STATIC_ROOT or STATIC_URL set at all. Also, no need for the s3utils.py file. These details are handled by the storage system automatically. The trick here is that I needed to reference this unknown path in my templates correctly and dynamically. For example:

<link rel="icon" href="{% static "img/favicon.ico" %}">

That is how I address my favicon which lives locally (pre-deployment) in ~/Projects/my_app/project/my_app/static/img/favicon.ico.

Of course I have a separate local_settings.py file for accessing this stuff locally in dev environment and it does have STATIC and MEDIA settings. I had to do a lot of experimenting and reading to find this solution and it works consistently with no errors.

I understand that you need the static and root separation and considering that you can only provide one bucket I would point out that this method takes all the folders in my local environment under ~/Projects/my_app/project/my_app/static/and creates a folder in the bucket root (ie: S3bucket/img/ as in the example above). So you do get separation of files. For example you could have a media folder in the static folder and access it via templating with this:

{% static "media/" %}

I hope this helps. I came here looking for the answer and pushed a bit harder to find a simpler solution than to extend the storage system. Instead, I read the documentation about the intended use of Boto and I found that a lot of what I needed was built-in by default. Cheers!

score 0 · Answer 5 · answered Feb 27 '20 at 07:35

If you want to have subfolders even before media or static seperations, you can use AWS_LOCATION on top of bradenm answer. Reference: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#usage

AWS_STORAGE_BUCKET_NAME = 'bucket_name'
AWS_LOCATION = 'path1/path2/'

Super Kai - Kazuya Ito · Answer 6 · 2021-03-30T05:18:11.457

Bradenm's answer is outdated and doesn't work so I updated it in March 2021.

Updated One:

Create a s3utils.py in the same folder of "settings.py":

from storages.backends.s3boto3 import S3Boto3Storage

StaticRootS3Boto3Storage = lambda: S3Boto3Storage(location='static')
MediaRootS3Boto3Storage  = lambda: S3Boto3Storage(location='media')

Then, add 2 lines of code to settings.py and change "myproject" to your folder name:

DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto3Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto3Storage'

The updated one has multiple "3s" as I emphasize below.

s3utils.py:

from storages.backends.s3boto"3" import S3Boto"3"Storage
    
StaticRootS3Boto"3"Storage = lambda: S3Boto"3"Storage(location='static')
MediaRootS3Boto"3"Storage  = lambda: S3Boto"3"Storage(location='media')

settings.py:

DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3Boto"3"Storage'
STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3Boto"3"Storage'

Check and compare with Bradenm's (outdated) answer.

"I respect Bradenm's answer."

How to set-up a Django project with django-storages and Amazon S3, but with different folders for static files and media files?

6 Answers6

Updated One:

Linked