What are the speed drawbacks from using multiple servers to serve a website?

Question

I know these questions are hard to answer :) But I want to get some kind of an idea of the impact of separating services onto different servers.

Specifically, I am setting up a website that will have a media server (apache serving static files), a app server (apache processing php files), a master db server and a slave db server.

I was going to run memcached on both the media and app servers as a shared pool, but was wondering if it was expensive (in terms of time and resources) to do so? Obviously everytime a memcached call is requested (from the app server) it has to make a tcp connection to the media server, work out where the results are and then return them.

For a small website setup like this, would it be better to bump up the ram on the app server and NOT pool memcached across servers? Or is the difference so small its not worth worrying about?

Even though I used memcached as the example, I would prefer if answers are kept fairly generic as the same scenario could be applied to other services (like a db).

Answer 1

If it is a small website, I do think this level of optimization is overkill.

But there is one way to find out for sure: test it.

We have used WCAT in the past to answer questions like this in the past. It's a great tool for seeing how the site will perform under various loads. JMeter is another great tool for something like this.

Using WCAT, for example, we ended up deciding to get a separate Hyper-V server to host our DB VM separate from our web app VM (in this case, Fogbugz). The test showed that even with a low number of concurrent users, having the DB VM on the same machine as the app VM made the application unusable (CPU was the bottleneck).

Answer 2

Need a few questions answered to better answer yours.

• Is it a single web site?
• Is it using intensive resources (heavy server side programming, streaming video, etc)?
• Are you just using video, db and php/.net?
• How well spec'ed are the servers?
• What speed are the hard drives and are they in a raid config?
• How many processors?
• How much RAM?
• Are you running 100mb or 1000gb nics and switches?
• What is your upstream speed?
• Is your OS streamlined and optimized?

Answer 3

Not nearly enough information give an answer with any likelihood of being correct, and actually running some tests/benchmarks would be the only way to be certain you're getting the right answer.

My best guess from what you've described is that pooling would increase the latency of an individual request, but increase the aggregate load you could handle before things bogged down.

But when I say that I suspect it might increase the latency of an individual request, I mean by an amount between some fraction of a millisecond to maybe a few tens of milliseconds, depending on your network setup. If your site is busy enough, this will be greatly outweighed by the speed advantages of making fewer DB calls with twice as much cache. If the media server and the app server are roughly equal hardware, it's entirely likely that your app server will get a high amount of CPU usage and the media server's CPU will be fairly idle, meaning you might actually get the best performance using memcached on just the media server and not the app server.

The only way to be sure about an optimization is to test. Test, benchmark, profile, etc. Without testing you'll never know if you're making performance better or making it worse. Run end-to-end tests (in other words, simulated web clients making all the requests for files a real web client would) with a realistic mix of users (so you get the right mix of cache hits and misses), both for moderate loads and "we just got slashdotted" loads. Automate your test so it's easily repeatable. Test with memcached on both/all servers, only on one server, only on the other server, with more RAM on one server than the other, etc...