Django InMemoryUploadedFile and asynchronous tasks

Question

I have a django application on which I upload several files of huge size. Once in my view, I want to do an asynchronous task on those files:

def my_view(request):
    Thread(target=_my_task, args=[request.FILES]).start())
    return redirect(my_url)

The problem is, by the time I uses the files in my thread, the main request has finished and the request object is deleted, alongside the InMemoryUploadedFile objects contained in it, and I get an IO exception:

ValueError: I/O operation on closed file.

How can I force the persistence of those files without writing them in my filesystem ?

That is one of the reasons why you don't do async tasks with threads in this way. use a proper task queue. — , Jul 03 '18 at 11:02
I understand using service like Celeri would be better but how does it help with the file closure problem ? — Charles-Edouard Lecat, Jul 03 '18 at 11:25

score 1 · Accepted Answer · answered Jul 03 '18 at 12:01

How can I force the persistence of those files without writing them in my filesystem ?

Short answer: you can't.

Longest answer: you could pass the files contents instead of the file objects, but this is unsafe at best (if anything goes wrong you loose your data) and cumbersome (when you'll switch to a proper task queue - threading is NOT the proper ways to run async tasks in Django - you'll find out that passing mega or gigabytes of binaray data as tasks argument doesn't work well, to say the least).

TL;DR: save your files to disk one way or another (preferably on your filesystem, that's what it's for) and use a proper async task queue.

Django InMemoryUploadedFile and asynchronous tasks

1 Answers1