I am building an application that deals with a lot of images and we need to process every image to apply various transformations such as scaling/resizing, filters, etc. How should I build that in such a way that it can scale as we grow and I won't need to go through the growing pains like Instagram and others had to do?
BTW, we're using ImageMagick right now process the images.
I was answering this in another thread too, but here's one way to do it without having to worry about scaling:
You could use a service like IronWorker to do image processing in the background and take the load off your servers. Since it's a service, you won't need to manage anything or set anything else up and it will scale with you as you grow so if you can do one image with it, you can scale to millions of images with zero effort.
Here's an article on how to do a bunch of image processing transformations:
Everyone has trouble scaling when they grow as fast as Instagram.
My general advice would be to use a work queue system to distribute the processing of images across n workers so that you can easily add more workers to pull jobs off the queue faster.
If you use this with storage of the images someplace like amazon s3, and can add more servers on ec2 for more workers (say using puppet, chef, scalr, etc. to fire up more worker boxes), you should be able to scale nicely.
