Django-storages S3Boto3Storage makes HEAD and GET requests on read

Question

I am using the S3Boto3Storage storage backend from the django-storages package in my Django application. I use the backend to handle large amounts of files in S3.

When reading a file from the storage, the backend makes separate HEAD and GET requests. I am doing this operation many times, so I would prefer to skip the HEAD requests if possible.

My read operation looks similar to this:

class MyModel(models.Model):
    img = FileField()

instance = MyModel.objects.filter().first()
instance.img.read()

I am using the default preload_metadata flag, which is False.

Is there another setting that controls this behavior of the read method?

My guess is that the HEAD request checks if the file exists before getting the file content. So maybe the HEAD call could be replaced with a try/except statement. But I could not figure out how to do that.

score 0 · Answer 1 · answered Jul 22 '19 at 21:35

ATTENTION: Following solution is not for production, your lead will punish you if saw this in your code :)

My packages version are old:
django-storages==1.1.8
boto==2.38.0
boto3==1.2.1
But main idea (I hope) will be understood

Place following patching(or if you wish - class could be extended) on some "start" level of your application(on your taste).

from storages.backends import s3boto


def monkey_init(self, name, mode, storage, buffer_size=None):
    """Only 'validate=mode != "rb"' was added in 'get_key' method's call"""
    self._storage = storage
    self.name = name[len(self._storage.location):].lstrip('/')
    self._mode = mode
    self.key = storage.bucket.get_key(self._storage._encode_name(name),
                                      validate=mode != "rb")
    if not self.key and 'w' in mode:
        self.key = storage.bucket.new_key(storage._encode_name(name))
    self._is_dirty = False
    self._file = None
    self._multipart = None
    # for files larger than this.
    if buffer_size is not None:
        self.buffer_size = buffer_size
    self._write_counter = 0


s3boto.S3BotoStorageFile.__init__ = monkey_init

In my version of django-storages call of method get_key doesn't pass validate variable into it. And I didn't find any way to force/change it except explicitly redefining __init__ method.

PS: If your setting doesn't containt explicit boto section in logging -> logs will not be shown for GET/HEAD/etc. requrests from boto. Just add boto section into logging setting part.

'boto': {
    'handlers': LOGGING_HANDLERS,
    'level': 'DEBUG',
    'propagate': False,
},

Django-storages S3Boto3Storage makes HEAD and GET requests on read

1 Answers1