0

At the moment I have a single server with 4 1TB hard disks, daily I haver over 150 MP3 music files uploaded (around 80mb each).

At busy periods there is over 300 people streaming / downloading these mixes all at once, 75% of the activity is on the most recently uploaded stuff which is all on a single hard disk.

My read speads on the hard disk are very low due to such high activity of 200+ reads all happening at the same time on a single hard disk (ran some tests with HDTach).

What would be a logical solution to solve this, a couple of ideas I had are:

  • Load balance with another server
  • Install faster hard disks (what are best these days? SCSI / SATA)
  • Spread the most accessed files over the 4 drives so it is sharing the load between all 4 disks, instead of all the most accessed (most recent) all on the most recently installed drive.

Obviouslly load balance is the most expensive option, but would it dramatically help?

Some help on this situation would be great!

Paul Hinett
  • 1,205
  • 3
  • 11
  • 19
  • 2
    How are the 4 drives setup? Are they in a RAID of some sort or are they individual drives? – Zoredache Mar 12 '10 at 18:28
  • They are not setup in RAID currently, simply just 4 hard disks. We do actually have an identical server on the same switch in the datacenter which is being used as a file backup server...for an extra $250 per month our provider said we can get this load-balanced with our primary server....do you think this could fix our issues? – Paul Hinett Mar 13 '10 at 03:46

4 Answers4

1

Your best bet will be to get some sort of RAID setup so that the load is spread across more than one hard drive. Also this will allow you to survive a failure of on of the disks in the RAID array without losing any data.

Based on what you've said a 4 disk RAID 5 array should work just fine for you as most of the data is reads.

Dennis Williamson
  • 62,149
  • 16
  • 116
  • 151
mrdenny
  • 27,174
  • 4
  • 41
  • 69
  • I was also going to suggest a cache on the RAID card, but for reads of large media files this might not help a bit. Although a BBWC (battery-backed write cache) might help if you're doing lots of upload. Are you using straight-up SMB for filesharing, or are you using some sort of streaming media server? If the later, it may have some options for caching, prefetch based on likely trands, or other optimization. – mfinni Mar 12 '10 at 20:47
  • No RAID setup unfortunately...although i beleive we should have it would be quite expensive to setup with the amount of storage we need. The files are served straight up from the local machines hard disks without any kind of streaming server. – Paul Hinett Mar 13 '10 at 03:49
  • You can't load-balance Windows fileserving across multiple servers, and I'm pretty certain you can't do it with NFS either. Adding RAID is what you should do, since you know the problem is that all of the IO is coming from the most recently-written disk. – mfinni Mar 14 '10 at 12:14
  • What do you mean by load-balance windows fileserver. Is what i want to do is have 2 identical machines running on the same network, both with 4TB storage which mirror each others data. I beleive this has the added benefit of server redundancy as well as sharing the load between 2 servers...am i correct in thinking this? – Paul Hinett Mar 14 '10 at 19:30
  • Yep that's feasible. You could also use DFS to replicate the file share contents between both servers. The only question left after that is how you redirect client requests to each server, to spread the load. Could be as simple as alternating between HTTP links when a page is generated. – Chris Thorpe Mar 23 '10 at 20:44
0

So what you want is a load balancer to ensure that both machines can serve requests, if needs be, and RAID to improve reliability and balance file read/writes over their disks. Oh and a rsync type tool to ensure the files are synchronised.

As for whether the various parts of this are "worth it", instead of guessing you need to do performance monitoring to see where the bottlenecks are. If disk storage is definately the issue then RAID is worth it. If serving network requests is the issue then server load balancing is your friend.

Rob Moir
  • 31,884
  • 6
  • 58
  • 89
  • I have now setup load-balancing which has helped lots, provided peace of mind and gave us lots more usable bandwidth (they offer 100TB of usable bandwidth with every server!!), however my next step is to remove the IO bottleneck on the SATA hard disks by introducing RAID. I beleive it would not be worth the extra cost of setting up RAID 10, but just offer disk striping for faster access. – Paul Hinett Mar 23 '10 at 10:54
0

How much memory do you have in the server? Perhaps using a ramdisk is an option if the system is devoted to serving files. Otherwise monitor the file cache counters in perfmon such as Cache - Cache Read Hit%, and the disk counters (in particular avg disk queue length, this should generally be around 3 or less.

Here's a document that has a few optimizations that may help.

David Yu
  • 1,032
  • 7
  • 14
  • The server has 6GB of memory, however it doesn't even break a sweat when it comes to memory, memory usage is usually below 1GB because the files are streamed to the user in chunks rather than stored in memory. My disk queue length does average around 3-5. – Paul Hinett Mar 23 '10 at 10:51
  • It's quite likely your drives/controller are a bottleneck, have you also checked your NIC utilization? If your NIC is fine, then you may want to look at using RAID to speed up your disk access, or trying to optimize your use of RAM through cache optimization or a ramdisk. – David Yu Mar 23 '10 at 19:19
0

You can scale in a few different direction here depending on your budget and hardware availability. Consider these steps, probably in this order but whatever suits your situation:

  • Re-jig the disks into a striped RAID set to increase raw disk performance. This'll mean backing up the files from the disks, creating the array, then moving the data back onto the stripe. If you can afford to lose some capacity for the sake of greater fault tolerance, go for a striped & mirrored array which will survive a disk failure

  • Add a dedicated RAID disk set for the server's page/temp file storage. Make this a fast RAID0 setup with a focus on performance. Ensure your app throws the files into memory when serving them, and that the page file is large enough to accomodate the files. This is a fairly cheap, straightforwards improvement.

  • Add more RAM to the server (it's relatively cheap stuff nowadays) with a focus on the bulk total RAM, rather than pure performance. The server can probably handle several different RAM speeds - and the slower stuff may be cheaper per Gb - You don't really need uber high RAM performance, rather you need bulk.

  • Re-tier the application so that the files are served from a dedicated host, and the rest of the web/database traffic and processing is handled by a seperate machine. This'll leave one server dedicated to file serving and you'll gain a performance benefit from this.

  • Tier your storage and add a fast stripe of disks to serve the newest MP3s from. Move files over to the slower storage once demand drops off. Feasibility of this approach depends on the effort involved in moving the files regularly, and how much of it you can automate in your app.

  • Add an array of SSD drives to serve the newest files.

Ultimately you can target a tiered storage solution that stacks up like this, in order fastest to slowest: - RAM - Pagefile - SSD - Fast SATA/SAS stripe - Bulk Disk array

Chris Thorpe
  • 9,953
  • 23
  • 33
  • I am going to accept this as an answer because it offers the most solutions to the problem, some of them I am already doing and some I did have planned. Thank You! – Paul Hinett Mar 23 '10 at 10:58