I have a django v3.0.2 site (python v3.6.9) where I upload an image, process the image, and then display it on a web site. By process the image, I create different sizes of the the one image, find and identify faces, ocr any text and translate it if needed, use several different algorithms to calculate how similar each image to the others, and other image processing algorithms. I use celery v4.3.0 and redis v4.0.9 to do all the calculations in the background in different celery tasks.
The issue I am having is that when I try to upload and process more than ~4 images in a loop, I run out of memory and the calculations start to throw exceptions. By run out of memory, my 5 GB of swap and 16 GB of ram are all used up and chrome even crashes. In my database, I keep a log of the progress of each task and whether it completes successfully or not. I can see many errors and many tasks not started.
If I put a time.sleep(10) in the loop that uploads the images and right after the celery workers are launched for each image, I can upload 60+ images in the same loop without any exceptions or crashes or errors in my task log.
I googled for some way to throttle the workers in celery so I don't have to use sleep, and found that perhaps celery does not have this capability Celery rate limiting with multiple workers in the celery issues forum. I found this article that might help Celery throttling - setting rate limit for queues.
Before I start building something on my own like the reference above, I thought I would ask the collective consciousness if there is a better (right?) way to throttle the celery workers than time.sleep(10) so I don't run out of memory and generate a lot of task errors.
There is too much code to post here, so here is an overview of what I am doing.
admin.py
def save_model(self, request, obj, form, change):
if form.is_valid():
if not change:
files = request.FILES.getlist('storage_file_name')
for f in files:
obj = Document() # model for each image
# add some data to the model
obj.save()
time.sleep(10)
doc_admin_workflow(<some parameters>)
else:
# some different housekeeping on the model fields
doc_admin_workflow(<some different parameters>)
super().save_model(request, obj, form, change)
def doc_admin_workflow(<some parameters>):
if not change:
# Decide which of the image processing steps to perform.
# Each processing step is a celery task of the form task_name.si(params)
# in a list called 'jobs' ~ 3-5 tasks
transaction.on_commit(lambda: chain(group(jobs),
change_state_task.si(document_id, 'new')).delay())
time.sleep(10)
else:
# decide what other processing steps (i.e. celery tasks) to perform
transaction.on_commit(lambda: chain(step_1, step_2, faces_2, ocr_job,
change_state_task.si(document_id, 'ready')).delay())
time.sleep(10)
All of the celery tasks used above are self-contained in that they do not pass data to the next task nor interact with each other in any way. They just calculate and read/write the database.
Also, these uploads are done by an admin to the site. Users interacting with the site do not upload images to the site nor use the image processing celery tasks. So, the time.sleep(10) is a possible solution, but it seems like a huge cludge and not very robust.
Thanks!
