Part of an application I'm writing allows for users to upload images which I then resize and automatically upload to amazon s3. Currently the image resizing is happening right in the view and I'd like to offload this via celery to distributed workers. My question is whats the best way to get the image to the worker. My current thinking is to store the image directly in the database and then just pass the id to the worker and have it retrieve it. Is there a better practice then temporarily storing it in the database until it can get processed?
Asked
Active
Viewed 4,030 times
2 Answers
4
As stated in the celery docs, it is better not to pass the whole thing (image) as an argument, for this will cause extra overhead. So it is better to save it in first place, then pass the photo id as an argument, retrieve the image into the task and do the resizing / uploading.
hymloth
- 6,869
- 5
- 36
- 47
-
1But is there any other option then saving it directly into postgresql? – whatWhat Mar 05 '12 at 22:40
-
Have a look here http://stackoverflow.com/questions/4330719/django-celery-how-to-send-request-filesphoto-to-task – hymloth Mar 06 '12 at 08:02
1
You can't pass the actual image to Celery because the image cannot be pickled, however you can deconstruct the image and send all it's data to a celery worker, where you can easily rebuild the image and then do whatever you want with it.
I build a small app which offloads the saving of an image to a worker, it would not be too difficult to fork it and add you own resizing to it. Check it out https://github.com/gterzian/django_async
gterzian
- 535
- 6
- 5