1

I currently have a dedicated web and database server running in datacenter (DC) 1. Im about to expand with yet another web server in datacenter 2 that will talk to the db in DC 1.

The latency between the two datacenters is about 2-5 ms. Is that appropriate for a high-traffic website (400.000+ visitors a day)?

MDMarra
  • 100,734
  • 32
  • 197
  • 329
Anders Vindberg
  • 161
  • 1
  • 3

4 Answers4

4

Maybe(TM). It certainly won't be helping your performance, and I'd be wary of having my machines spread out like that just from the potential failure modes it introduces, but it's probably not your biggest performance problem.

On the other hand, it's amazing what hosting providers are capable of if pushed, so I'd tell your provider that their proposed solution is unacceptable for your needs, and if they can't provision another server in the same network as your existing machines, then you'll take your business elsewhere. Chances are they'll miraculously find some space close by.

womble
  • 96,255
  • 29
  • 175
  • 230
  • 1
    Not sure about the last paragraph; asker never mentioned _why_ he's moving to a second data center. – gravyface Mar 16 '12 at 09:12
  • Im trying to keep the cost down. There are still room in DC 1, its just more expensive :) – Anders Vindberg Mar 16 '12 at 11:44
  • 1
    @AndersVindberg in that case I'd suggest either moving your existing server to DC2 so it's alongside your new one; or duplicating your DB in DC2 as well and configuring a load-balancing/failover system. – Dan Is Fiddling By Firelight Mar 16 '12 at 12:38
  • @gravyface: I've never had equipment in a second DC so close by without being forced to by the provider. I can't imagine why you'd ever want to do so, given the downsides (latency, increased chance of failures due to interconnect complexity) without any upsides (anything that'll take out DC1 will probably do for DC2, as well). If cost is an issue, then as Dan says, you move all your equipment to the cheaper facility, and save yourself money *and* latency *and* complexity. I didn't realise that was news to anyone. – womble Mar 16 '12 at 21:27
  • @womble I completely agree: I've only ever setup a distributed data center for geographical redundancy (DRP) reasons, never because adding another half rack in the same data center was prohibitively expensive. If that was ever the case, I'd bail in a heartbeat; there's just no reason for that given that colo facilities are a commodity business nowadays. – gravyface Mar 17 '12 at 03:27
2

2-5ms is not too bad imho. Then again, it also depends on how your application handles database calls.

More interesting though, might be looking at fault-tolerance, you might want to set up an identical solution in DC2, so you have a DB server in both, that way you can quickly recover from any disaster in one datacenter. A 2-5ms delay would be very appropriate for database replication, and if you use mySQL, you might even be able to set up multimaster replication and get better performance between web servers and nearest DB server

Though there are nothing wrong with pushing your service provider to commit to an acceptable level of performance, you also need to "face the facts". If the Datacenters are 300km apart, the first 2ms is simply due to the fact that not even fiberoptic WAN lines exceed speed of light (roughly 300.000km/s). The ping time is a roundtrip time, so 300km would in fact mean the time taken to travel 600km (300km each way). That takes 2ms at the speed of light, and no hosting provider can tamper with a physical constant like the speed of light

Mathias R. Jessen
  • 25,161
  • 4
  • 63
  • 95
  • Smart caching can often reduce ping to 0 :) – orlp Mar 16 '12 at 10:38
  • Interesting point on having a complete copy of the entire website and database in DC 2 if something were to happen to DC 1. I think that will be my solution. The server provider can only provide a failover IP (no NLB) which further supports this solution. – Anders Vindberg Mar 16 '12 at 11:39
2

It isn't really possible to answer your question -- you haven't provided any measurements of the impact on your site, and you also haven't defined what your performance goals are.

But one important pointer: Many webapp frameworks use a blocking reads strategy, i.e. while the webapp framework is making SQL queries, the webapp thread in questions is flagged as busy to the operating system. So you could see CPU use on your frontend servers increase significantly because of the added SQL query latency.

Some webapp frameworks like ASP.NET MVC provide an asynchronous I/O interface, which could alleviate the above problem. But async I/O is not used by default, and it would almost certainly be too expensive to re-architect your site code to use this.

Why not just migrate your old servers to the new DC, and set up a nice private VLAN for all your servers in the new DC?

  • Thanks for this insight! The website isn't using the async keyword (C#, MVC), but I do use a substantial amount of caching. However, around half a million search queries a day cannot be cached. – Anders Vindberg Mar 16 '12 at 11:43
1

That seems rather high to me, and while I've had databases on different networks, they've always been in the same physical location, so latency wasn't an issue. Depending on how many queries per request you're servicing on average (and how little caching you're doing in the Web/application tier), that could really start to add up. I would want to test that thoroughly while simulating load that's appropriate for your usage patterns.

Is there a reason why you're setting up a Webserver in a different data center? If it's for redundancy, perhaps setting up a replication scheme to another local database at Data Center 2 would be more appropriate.

gravyface
  • 13,957
  • 19
  • 68
  • 100
  • From what I can extract in your answer, I believe the latency is too high with the amount of requests we are experiencing. Thanks – Anders Vindberg Mar 16 '12 at 11:48
  • @AndersVindberg I think all of the answers are really saying, "it depends" and "test vigorously". You may find that 2-5ms is _fine_, but without a baseline prior to this new remote database, and following up _after_ moving it with test cases, it's impossible to answer yes or no; we're all just going by our "gut" (past experiences). – gravyface Mar 16 '12 at 17:27