django-storages + sorl_thumbnail + S3 not working well together (URLs mismatch)

Question

I am using django-storages and sorl_thumbnail together and I am using Amazon S3 for static and media files. I am using one bucket with 2 folders, 1 for static and 1 for media.

Here is my config:

MEDIA_ROOT = '/media/'
MEDIA_URL = 'https://s3.amazonaws.com/my-bucket/media/'
STATIC_ROOT = '/static/'
STATIC_URL = 'https://s3.amazonaws.com/my-bucket/static/'
AWS_STORAGE_BUCKET_NAME = 'my-bucket'
DEFAULT_FILE_STORAGE = 'my_lib.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'my_lib.s3utils.StaticRootS3BotoStorage'

MediaRootS3BotoStorage and StaticRootS3BotoStorage are defined like this:

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')

When I am using sorl_thumbnail, the thumbnails generated are located in the right directory: https://s3.amazonaws.com/my-bucket/media/cache but when sorl_thumbnail is trying to retrieve an already existing thumbnail, the URL generated is: https://s3.amazonaws.com/my-bucket/cache, you will notice that the media folder is omitted.

Do you have any idea how I could fix that?

I know I could just use django-storages and have my static and media files all mixed up in my bucket, but that's a bit too dirty for my taste :)

Thank you!

score 17 · Accepted Answer · edited Feb 13 '14 at 15:50

I was able to make it work by defining MediaRootS3BotoStorage and StaticRootS3BotoStorage as follows:

from storages.backends.s3boto import S3BotoStorage
from django.conf import settings

class StaticRootS3BotoStorage(S3BotoStorage):
    """
    Storage for static files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'static'
        super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)


class MediaRootS3BotoStorage(S3BotoStorage):
    """
    Storage for uploaded media files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'media'
        super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)

This link can be helpful https://github.com/jamstooks/django-s3-folder-storage

The `super()` methods on each of the `__init__()` methods above are calling the wrong classes. They should be: `super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)` and `super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)` — niceguydave, Jun 19 '13 at 09:14

score 8 · Answer 2 · answered May 23 '13 at 15:45

I had the same problem and the solution by Salma Hamed turned out to be the right one for me.

Before we had

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')

which resulted in the wrong 'storage' values in our thumbnail_kvstore table. This lambda definition does not create a new class and thus type(StaticRootS3BotoStorage()) returns 'storages.backends.s3boto.S3BotoStorage', which is written into the table. Because these 'storage' values are used to instantiate later the storage in order to get the image URLs when displaying, this resulted in S3BotoStorage() to be used for this. So the 'location' argument was lost.

The solution by Salma Hamed that defines these custom storages as classes fixes this.

Thanks for that!

Do you remember where you learnt to use the previous solution of `lambda: S3BotoStorage(location='static')`? I would like to go post a comment or correct it. — Flimm, Oct 31 '16 at 19:47

score 1 · Answer 3 · answered Oct 05 '12 at 17:58

1

Have you tried setting THUMBNAIL_PREFIX to media/cache/?

http://sorl-thumbnail.readthedocs.org/en/latest/reference/settings.html#thumbnail-prefix

answered Oct 05 '12 at 17:58

jasisz

1,288
8
9

2

Yes, I have already tried, thank you. But it doesn't work either, the problem will be the same, for example if I set `/media/cache` when the files are pushed the url will be `/media/media/cache`, wand when retrieved `/media/cache` – e-Jah Oct 06 '12 at 04:43

score 1 · Answer 4 · answered Oct 13 '12 at 04:17

1

I had this same exact problem but I figured out a way around it.

I set my DEFAULT_FILE_STORAGE back to storages.backends.s3boto.S3BotoStorage, that way, when it looked for cache/ it would not miss, and I could still upload all of my files to media/, and python manage.py collectstatic still works properly because I still have that set as StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static').

Hope this help you, because this problem was driving me crazy.

answered Oct 13 '12 at 04:17

Derek Parker

81
4

thank you for your help! But the result for me when I use these settings are the static files go the the /static/ folder on S3 but all the uploaded files are in the root folder, they don't go in /media/. Any idea? – e-Jah Oct 13 '12 at 09:30
Sorry for the late response, but make sure that in your models you are using `UPLOAD_TO='media/'`. That should put your uploaded files where you want them. – Derek Parker Oct 16 '12 at 21:54
Yeah, I guessed that that you were doing, but I would prefer not, else it just breaks the purpose of how django works :) Thank you anyway! – e-Jah Oct 17 '12 at 01:51
@e-Jah did u find a solution, coz I am facing the same problem – Salma Hamed Nov 08 '12 at 14:23
@SalmaHamed: unfortunately my solution has been to switch to my own server(s), it fixes so much trouble! – e-Jah Nov 08 '12 at 23:43
@e-Jah did u switch the amazon s3 or ur main server or all of them :) ??. Because my current combination of server(S) are: Heroku + Amason s3 because personally I thought, it has nothing to do with the server. and it's a sorl-thumbnail url translation problem ?? – Salma Hamed Nov 09 '12 at 07:16

score 0 · Answer 5 · edited Jan 27 '13 at 08:46

Found that sorl-thumbnail is returning the cached KV image url using STATIC_URL (on the next request after the initial thumbnail is created). Appears MEDIA_URL has no affect.

Not the best solution. Added a S3 routing rule.

<RoutingRules>
  <RoutingRule>
    <Condition>
      <KeyPrefixEquals>cache/</KeyPrefixEquals>
    </Condition>
    <Redirect>
      <ReplaceKeyPrefixWith>media/cache/</ReplaceKeyPrefixWith>
    </Redirect>
  </RoutingRule>
</RoutingRules>

score 0 · Answer 6 · answered Jul 13 '20 at 06:11

IDK why but @SalmaHamed 's solution didn't work for me (Maybe its because of change in the django version or something). Instead, I replaced: MediaRootS3BotoStorage = lambda: S3Boto3Storage(location='media') with: class MediaRootS3BotoStorage(S3Boto3Storage): location = 'media'

django-storages + sorl_thumbnail + S3 not working well together (URLs mismatch)

6 Answers6

Linked