What Sort of Server Setup Am I Likely to Need? - School A/V streaming

Question

My prior experience with servers has generally been limited to home file-sharing servers, low-traffic web-servers, and the like. This leaves me with the technical knowledge of how to set up a system, but little experience in terms of scaling said system.

My current project, however, has me as the technical lead in setting up a school for online audio and video streaming. The difficulty I'm running into is that I don't quite have the experience to guess what they'll need, and they don't have the experience to tell me - so I've tried to ask as many pertinent questions about what they want to do with their server, and here's what I found out:

• About 1000 simultaneous users, and hoping to expand (possibly significantly)
• Both video and audio streaming, at obviously the highest quality possible
• Support for both live and playlist-based streaming.
• Probably only one channel, but as it's an educational opportunity, I imagine letting them have a few more wouldn't hurt. (strike this - I thought that channels was referring to streams, which is odd, given that I'm originally an audio guy!)
• No word on whether they're locked into Windows or whether Linux is acceptable.
• Approximate budget - $7000. It may actually be about $2k less than this, because of a mishap with another technology firm (they ordered a $7000 DV tape deck for some reason, and now the company wants them to pay a 30% restocking fee).

The tentative decisions I've already made:

• I'm planning on using Icecast 2 for my streaming server, fed by VLC Shoutcast encoding.
• Since the school already has a DMZ set up, I plan on placing the Icecast server in there, and feeding it through their intranet from a simple workstation computer in their studios.
• This system isn't in any way mission critical - it's an education tool (they're a media magnet school), so I figure redundancy is not worthwhile to them from a cost:benefit perspective.

What I don't know is this:

• How powerful of a server will I need?
• What is likely to be my major throttle - bandwidth? How can I mitigate that?
• Will I need anything special for the encoding workstation other than professional video and audio capture cards and a copy of VLC?
• Are there any other considerations that I'm simply missing?

Thanks a lot for any help - if there's more information you need, let me know and I'll tell you all I can.

---

Edit 1:

I'm expecting to go out either in OGG or MP3 - probably MP3 as it's more widely supported (and more familiar to parents of grade-school students.

The content will consist of live/prerecorded standard definition video packages (likely in Quicktime, for the latter). Audio-only broadcasting will likely consist of MP3s, transcoded down to a more manageable quality. DRM is not an issue, nor is copyright (at least, copyright is not my issue). Content is generated whenever the students feel like broadcasting something, but the storage of this media shouldn't be a problem. They're not generating raw movie footage here - they're kids doing small TV packages - I'll throw 4 Terrabyte hard drives in RAID 10 on the workstation and call it a day.

Much of the rest doesn't quite seem to apply. The most relevant portion is the inherent limitations on the network. When I tried to inquire about what those might be, they really had no idea - as they've never really stress-tested it. Their internet connection is used largely for standard internet service for the students and teachers, and serving a website likely built in the 90s. I doubt they'll go through an expensive refitting of their NAT and front-end systems - the kids will simply have to live with what they have.

I don't think my server(s) will have to worry about massive caching, or large chunks of data storage - they will simply receive the stream ready to go from the internal workstation/source client, and have to replicate and distribute the stream to the clients. I expect to run into bandwidth bottlenecks before I find myself clamoring for more memory.

Regarding multicasting - I thought the internet at large didn't support it - and external clients make up the large bulk of our potential subscribers. My big question is a bit more rudimentary - what is a "decent server"/server system for this sort of application? I'm having difficulty envisioning what quantity of machines I will need, as well as what they need to be capable of. Any thoughts? Thanks in advance.

Answer 1

I build IPTV systems and you're not going to like it but you've got a very tough task ahead of you, especially considering your budget.

Let's go through it step by step (I'm aiming this answer to a broader audience than just you if you don't mind);

• The first thing anyone building such a system has to do is define its customers in terms of locations, OSs, players and ideally understand the networking aspects between clients and servers - you've done a lot of this already. This is important as it defines the server code, codecs, bit-rates and whether you can use multicasting at all.

• The second thing to do is understand your content; where does it come from, in what format, how often, copyright issues, DRM requirements, how quickly it needs to be available and how long does it need to be retained for. I think you've got some of this covered but you can probably ask some more questions along these lines.

It's far from a simple task but based on the results of the two blocks of questions above you can start to design your system, again from the point of ingest down - how EXACTLY are you going to reencode (if required) your content, does it need QA'ing, where does it get DRM stamped and how are you going to deal with the risk of temporarily having unencrypted video if that matters. This ingest/staging system needs to be considered up front, you don't want to have to 'hand-crank' this process at all because it'll get repetitive very quickly.

Once you've figured out this system you need to know what's doing your content cataloguing and how the ingest system feeds into this, also how you're going to publish this catalogue and how you removed 'aged' content. This publishing system will then initiate client playback and may require some form of entitlement checking system, which brings its own problems that we could tackle another time.

Now we get onto the part you're directly interested in, the streaming servers.

By this point you'll know your user base data (who, what, where, when, how etc.) and this will help you calculate your peak bandwidth requirements (usually something like 'peak users' x 'highest bit-rate').

"Best quality" is something of a misnomer as I'm sure you don't mean "48-bit 8k HD @60fps" but more like SD quality or so, as a guide I do SD at around 1.5Mbps and HD at about 6Mbps but this will be defined by your codec and requirements. So as an example 1,000 users playing back 1.5Mbps of unicast traffic obviously equals 1.5Gbps, this is where the hard bit comes in. Firstly can your network, the actual trunks and switches themselves consistently deliver data of this nature? You'll need to sit down and work out where your weak-spots are. Can you QoS this data to protect it from someone pulling down a big download and killing an entire network segment's video? There's also the fact that if you're your going over 1Gbps on your NICs you need to ensure that you have them teamed in the right way so that the second and subsequent team member actually gets to carry it's share of the traffic, that or go to 10Gb NICs.

Then we're onto my specialist area, the storage - by now you've worked out how much storage you'll need for content at the start, how quickly it'll grow, it's maximum size and it's 'churn rate' (how much comes on and off the content store over what period). This will tell you how much storage you need, but if you have 1,000 users all looking at a good-size catalogue of content what happens is that you have to pay very close attention to caching and the random-read capability of your storage.

If you're VERY lucky and your server can cache 100% of your content then you're in luck, you just need a decent server, lots of memory and a 64-bit operating system. If you expect users to have access to a larger content store than you can cache then you need to ensure that your storage system can deliver your peak streaming requirements, say 1.5Gbps in the example above, consistently. How you do this depends on the size of content store and if you need more than one streaming server for this (as if you do you'll have to figure out if you're going to shard or share your video).

You can look at SSDs for example but don't look at the 'headline' numbers such as 500MBps, what you need are SSDs or disks that can be RAIDed to ensure that peak delivery speed. There are plenty of SSDs and disks out there that are great for workstation or low concurrency servers but when you hit them with a requirement to keep up with say 1,000 users all pulling little pieces of large files at different point in the files - well basically they don't keep up and neither do most caching algorithms - you have to know that your storage can do the work on its own if it has to.

If it's any consolation the live-streaming bit is much easier but probably deserves it's own machine to handle the capture/encryption/streaming rather than the main content-store based server/s. You're lucky in that you probably have a better understanding of your network as it sounds a closed system, if you can you want to ensure you can multicast this otherwise you're into the same bandwidth management issues I describe above but with a better cache-hit opportunity.

I hope this helps, like I say I've aimed it at a wider audience than just this question and I may come back and add/edit (once I've had my first cup of tea for the day!).

Answer 2

Stream bandwidth depends on the quality and the receiving end computing capacity. You could aim for 0.5 Mbit/second per client. This is aprox equivalent of DVD on DivX compression.

You should try finding a solution where server(s) don't need to do any coding or transcoding and the stream is already in the correct format. Especially not per client basis.

High end hardware costs proportionally more, and you end up in trouble if it brakes down. Try a farm of lowend sub-$1k servers with a spare one if needed. Using the same image for system disk for the servers is a good idea.

With only streaming data - no coding/transcoding - the bottleneck comes from the network interfaces (and possibly network infrastructure). You should calculate not to exceed 10% of named bandwidth of the interfaces or the connections will slow down.

I am not sure if your network everywhere is equipped to handle 500 Mbit/s additional traffic (with <10% capacity). If not, you could try doing servers for each location or cut down the bandwidth. You can use five servers with gigabit ethernet but they cannot be connected to a hub with a single gigabit uplink.

Hope this helps.

Answer 3

Contact the guys from the TightwadTech, this is right up their street, and hopefully we get a podcast out of it :-)