What kind of hosting is used for *tube sites?

Question

When a site grows from a just-a-fun project to a site with bigger load of visitor, and you want to enable them to upload videos, you might find yourself in a need of a better hosting, including dedicated server and a no-limit web traffic (or some reasonable limit).

So, if people can upload their videos, and if page has around 1000-10000 visitors per day, what kind of hosting is there to choose from? What is needed in that case?

Thx

score 3 · Answer 1 · answered Jan 10 '11 at 08:25

3

So, if people can upload their videos, and if page has around 1000-10000 visitors per day, what kind of hosting is there to choose from?

Shared hosting or small VPS. 10.000 users is not a lot per day. Come back when you have 500.000 visitors in an hour ;)

Seriously: larger sites colocate and have RACKS of servers. Look at the infrastrcutrue that runs slashdot or stackoverflow. This is not "ok, giveme some webspace" type of hosting. It is "where can I put my own servers" type of hosting.

answered Jan 10 '11 at 08:25

TomTom

51,649
7
54
136

Thx for edit/answer :). But, what happens with storage? 10.000 of users per day can upload a bunch of stuff. Shared hosting usually have quite low disk quotas. – playcat Jan 10 '11 at 08:37
Then start buing your own servers ;) Obviously then shared hosting is not for you. Expect to pay. – TomTom Jan 10 '11 at 10:27
Would you have a link for the Slashdot infrastructure ? – Antoine Benkemoun Jan 10 '11 at 14:41
1

Slashdot did some articles on it: http://slashdot.org/article.pl?sid=07/10/18/1641203 http://news.slashdot.org/article.pl?sid=07/10/22/145209 – Kjetil Joergensen Jan 11 '11 at 15:38

score 3 · Accepted Answer · answered Jan 10 '11 at 11:52

TomTom raises a good point when he says that 10,000 users is not a lot, but number of users isn't the important thing here in determining the "size" of your hosting requirements. It's a point worth remembering though: if you see 10,000 users as an upper limit then youtube's problems are not your problems.

Of those users, how many will be uploading and how many simply viewing the videos. The resources required to service both users are different, and you will probably have far more "viewers" than "producers".
You need to consider how the length/file size of the videos, not just the number - 1000 2 hour long HD videos will generate a very different problem for both storage and playback than 1000 10 min videos filmed on the average phone camera.
What processing will be done on these videos - e.g. a guest uploads a video - will it be converted to a different format for playback? That has implications for processor requirements.
Are you prepared to write your own code that is highly optimised for delivery of video the way you want it to be done, or will you be using "off the shelf" site code, e.g. just bunging it all into a CMS and hoping it works OK?

To a large degree, responsiveness of the site will be a big part of its success - users won't stay at a video site that doesn't play videos in a timely manner. So rather than saying "what kind of hosting will take my site, and how much will it cost" you need to say "This is what is an acceptable level of service for my site" and do tests to determine your needs from that (and accept that it may be complex and/or expensive). This would make shared virtual hosting possibly very problematic once your site starts to grow.

Once you get to a certain size then you will probably need your own dedicated servers.

I'm still far from that, but it's always a good thing to think ahead, even if I'll ask myself this same question in future (hopefully). Once again, thank for this answer, it's much appreciated. Also, since this answer is listed as ANSWER, I'd like to thank all the users participating in answering it. — playcat, Jan 12 '11 at 07:36
@Playcat it's always a good idea to design for where you are but with an eye to where you want to be, so that you don't make any decisions that make the journey more difficult. — Rob Moir, Jan 12 '11 at 08:51

score 0 · Answer 3 · answered Jan 10 '11 at 10:46

The short answer is that Amazon S3 really will survive most web-sites for a long while. There are many intermediate stages of a website, from one PC in a basement, a VPS somewhere, a real server somewhere, 3 or 4 real servers arranged for function, a small cluster with a load balancer up to the google style with multiple sites and a CDN.

The importance is not to over-provision as well as under-provision. You want to project your load in one years time and plan for that. Not all websites are about huge numbers of traffic - some are about quality traffic.

Youtube's infrastructure is very interesting and is enormous. There are many great articles about it online.

The only thing bad is that S3 is REALLY expensive for "base load" that you need to keep around all the time. REALLY expensive. Buying your own dedicated stuff is a lot cheaper. — TomTom, Jan 10 '11 at 10:51

score 0 · Answer 4 · answered Jan 10 '11 at 11:08

So, if people can upload their videos, and if page has around 1000-10000 visitors per day, what kind of hosting is there to choose from? What is needed in that case?

You'll need storage, and lots of ingress bandwidth and egress bandwidth, so use something symmetrical.

That's not an extremely high number of page hits, so just about any configuration of service will do you. (Cloud/VPS/Colo/rack space/etc).

score 0 · Answer 5 · answered Jan 11 '11 at 15:22

I would find that the hosting may stay near the same provided you already have steady reliable infrastructure where your servers are operating.

The only real change I would look into for this specific instance would be outsourcing the media portion to a premier provider of that service, I'd recommend taking a look into Brightcove. http://www.brightcove.com/en/ similar to a CDN such as Akamai.

Best of luck and congrats, sounds like a cool project!

score 0 · Answer 6 · answered Jan 11 '11 at 18:04

This is from 2008, but is still pretty relevant, High Scalability.com YouTube Architecture article they are in a very different scale:

YouTube Reaches One Billion Views Per Day. That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour.

Contrast to expectations of 10,000 a day! That article also has some info on how they expanded from a small, fun site to the monster they now are.

Not just one part of a shared server, they have clusters of servers serving individual videos.

And a set of tours inside one of Google's datacentres Cnet YouTube tour reveals Google data center designs. They're at the kind of level, where their datacentres have their own water-treatment facilities just to provide clean water to keep the servers cool.

Finally a surprising article from Data Center Knowledge on YouTube’s Bandwidth.

What kind of hosting is used for *tube sites?

6 Answers6