4

For my Django site, I'd like to:

  1. Accept images submitted by users
  2. Generate thumbnails from those images
  3. Put both the original images and the thumbnails onto separate, multiple servers that are dedicated to serving images

I need multiple, separate servers to serve the images/thumbnails to ensure I have enough IO performance.

What is the best way to build a distributed image serving system like this? Any open source software that would help?

Thanks.

Continuation
  • 12,722
  • 20
  • 82
  • 106
  • I've found one idea in comments of this post - http://stackoverflow.com/questions/3981080/form-uploading-files-to-different-server-without-following Yes, of course this is not answer, but probably nice idea to just upload files temporary to filesystem where Django installed and then send them to the servers where it should be... – Ignas Butėnas Apr 11 '11 at 03:49

1 Answers1

4

This sounds like a job for distributed task queues. My personal favorite is Beanstalkd b/c it is so lightweight and easy to use.

Server: https://github.com/kr/beanstalkd
Client Library: https://github.com/PeterScott/beanstalkc
Django Helper: https://github.com/jonasvp/django-beanstalkd

The way it would work is like this:
1. File gets uploaded and stored locally
2. Job gets created by upload view and put into a beanstalk tube
3. Beanstalk workers are watching the tubes for work, they grab the new job
4. The worker could create the thumbnails and distribute it to as many servers as you need

The beauty is that you can have as many workers feeding off the tube as is required. So if you ever get behind, simply fire up more workers. The workers can be on the same machine or on many separate machines, and it will all work fine.

rfadams
  • 1,898
  • 2
  • 19
  • 21