6

I would like to know what is the best solution for storing large amount of images on multiple servers like google, facebook.

It seems that storing in filesystem is better then inside a database but what about using a noSQL DB like cassandra.

Do Google/Facebooke store the same image in multiple servers for the load balancing. How does it work? What is the best solution?

Thx a lot

Community
  • 1
  • 1
Adam Paquette
  • 1,243
  • 1
  • 14
  • 28
  • Cassandra will work. You may also want to look at distributed file storage solutions like MogileFS, HDFS, etc. – DougW Jul 23 '14 at 17:37

3 Answers3

4

There's nothing wrong with the approach you're taking. As mentioned, there are caveats, however, the possibilities do exist, and a lot of people and companies are successfully storing files in Apache Cassandra.

  • zjffdu/cassandra-fs is the first solution i'd look into. Now, this was last developed 2 years ago, so I'd be a bit cautious on it working the first time, out of the box. Apache Cassandra is now at version 1.0.x, with 1.1.x on the way. 2 years ago, that was version 0.6.x maybe? A lot has changed & improved in 24 months.
  • semantico/cassandra-fs a fork ... last touched 7 months ago
  • favoritas37/cassandra-fs another fork ... last touched 3 months ago and indicates compatibility with the 1.0.5 branch of Cassandra

The principal behind this is to take a file, break it into a set of chunks and store those chunks as columns in a row. When retrieving, pull each column, reassemble the file and voila.

Cassandra FAQ: large file and blog storage

...files of around 64Mb and smaller can be easily stored in the database without splitting them into smaller chunks...

Lucene indexes in Cassandra

...its files are broken down into blocks (whose sizes are capped), where each block (see FileBlock) is stored as the value of a column in the corresponding row...

You'll get more positive feedback on the Cassandra mailing list and on the IRC channel.

Finally, this is from 2009, and written by folks at Facebook, which should go some way to help answer more of the fundamental questions you have: Cassandra - A Decentralized Structured Storage System.

sdolgy
  • 6,963
  • 3
  • 41
  • 61
1

Note, I know this is an old question, I just want to counter balance some misconceptions about cost as I'm doing this right now as a test.

Unlike what DavidB thinks, it does not cost millions - even if you were to run dedicated hosted hardware, you'd easily be under a couple thousand/month (BTDT, one of my clients is running a 8 node cluster for about $800/month). That said, that's a maintenance headache you want to avoid, and Cassandra on EC2 is far easier to deal with.

You could easily run a substantial production cloud on EC2 for less than $1000/month and you can do R&D clouds for less than $100/month (I spend about $52 last month for an 10 machine test cluster). I highly recommend using TurnKey Linux to manage & provision your R&D farm, as their tools will allow you to migrate instances from your desktop to pretty much any virtualized hosting platform in a few minutes (and vice versa). Plus they have really slick integration with EC2.

For really serious levels of traffic, Pintrest once stated they spend $15 to $50/hour depending on server load, auto-scaling to meet traffic demands, see http://www.theregister.co.uk/2012/04/30/inside_pinterest_virtual_data_center/ for details

The real cost is in setup and manage of your distributed Cassandra instance. Luckily, NetFlix has just release a ton of manage tools just for this. You can find them here: https://github.com/netflix - there are also a ton of interesting videos about NetFlix's use of AWS, particularly moving stuff from Cassandra to S3 - see their blog here http://techblog.netflix.com/2012/12/videos-of-netflix-talks-at-aws-reinvent.html

ckm
  • 1,326
  • 10
  • 15
-1

If you want to store in a "cloud" environment you're best going with a cloud solution that has the resources such as Google App Engine or Amazon Web Services. You're not going to be able to setup your own if that is the question. It will costs millions of dollars and resources to manage them. And yes, Google and Facebook use thousands of servers to distribute their data in "clouds".

DavidB
  • 2,064
  • 3
  • 17
  • 17
  • I don't wanna store into the CLOUD, I want to start with 1 server then 2......It's not gonna cost millions. I also want the load balancing. – Adam Paquette Mar 25 '12 at 02:31
  • You said you wanted to store the same image in multiple servers right? That is storing data in the "cloud". Technically it doesn't have to be on a third party architecture. You can roll your own cloud but don't underestimate the cost of doing even 2 servers. How are you going to be able to guarantee the consistency of those 2 servers? That would be extremely hard to roll your own. You could try or buy software specifically designed for it which isn't cheap. Then when you start scaling up from 2 servers that's where things get really fun. Are you looking to your own or purchase software for it? – DavidB Mar 25 '12 at 02:37
  • Cassandra could work for you if you don't mind investing the time in it. Every solution will have its nuances and headaches. – DavidB Mar 25 '12 at 03:13
  • 1
    This answer is nonsense. Multiple servers does not equal "The Cloud". Clusters and distributed storage existed long, long before marketing came up with that term. There are also many perfectly viable solutions for private distributed file hosting that can be set up cheaply. Cassandra, HDFS, MogileFS, etc. – DougW Jul 23 '14 at 17:34
  • I didn't say multiple server equals "The Cloud". I said Google and Facebook use thousands of servers to distribute THEIR data in "The Cloud". I think anyone who has been around the block in the last 20 years knows that "The Cloud" is just marketing. And "Cheap" does not mean you get the load balancing and data replication across geographic locations around the world that Google uses. That was my point. If you want a simple private solution then go ahead and set one up "cheaply". When I originally wrote this 2 YEARS AGO I thought the poster was trying to replicate a Google/Facebook system. – DavidB Aug 11 '14 at 05:31