Memory leak in Ajax implementation of a file uploader (Django on Heroku)

Question

I've created an ajax file uploader for Django, but each file uploaded takes up a big piece of memory (30-80 mb), and doesn't seem to let it go.

We're on Heroku, which allots 512mb of memory per dyno, so I quickly start getting over-memory errors.

Here is the Django view code to handle the request:

if request.is_ajax():
        # the file is stored raw in the request
        upload = request
        is_raw = True
        try:
          filename = request.GET[ 'add_image' ]
        except KeyError:
          return HttpResponseBadRequest( "AJAX request not valid" )
        (fileBaseName, fileExtension)=os.path.splitext(filename)

        uniquename = biz_id + "__" + get_a_uuid() + fileExtension
        saved = save_upload(upload, uniquename, biz)

And here is the save_upload code:

try:
    #BusinessImage is my Django model.  It uses django-imagekit to processs
    #the raw uploaded image into three sizes (plus the original)
    bi = BusinessImage(name=uploaded.GET.get("name"), business=biz)
    if not BusinessImage.objects.filter(business=biz).exists():
        bi.primary_image = True
    bi.original_image.save(filename,ContentFile(uploaded.read()))
except IOError:
    # could not open the file most likely
    return False
finally:
    uploaded.close()
return True

This code is adapted from this post (thanks to Alex Kuhl and Thunder Rabbit).

I'm thinking the memory problem might have something to do with django-imagekit or I may not be closing the files properly, but I'm not sure. I'd really appreciate any help.

Thanks!

Clay

I don't think this is directly related to django-imagekit. The leak also exists in django-ajax-uploader which is based on the same js code. See: https://github.com/GoodCloud/django-ajax-uploader/issues/12. — Dirk Eschler, Aug 03 '12 at 23:40
@DirkEschler -- really appreciate that link, Dirk. django-imagekit does have documented memory problems (see: https://github.com/jdriscoll/django-imagekit/issues/63), but even after migrating away from it I noticed I was still having issues, albeit less severe ones, and your link explains them. Thanks again. — Clay Wardell, Aug 06 '12 at 16:12

score 1 · Accepted Answer · answered Aug 27 '12 at 18:44

Django upload handlers are not generally for processing and trasforming files, they are for getting them ready to pass to views in the request.FILES array (via saving to /tmp or saving to memory).

Try using the [nginx upload module][1] (and upload progress) if you want an easy, quick, and low memory way to get progress feedback. It copies the upload to a specified location on disk and passes the request onto the view with the file path, size, and mime type in POST vars. Much more efficient than doing so with django.

Memory leak in Ajax implementation of a file uploader (Django on Heroku)

1 Answers1