Business Internet Outage/Disaster Recovery

Question

We have all of our corporate servers co-loc'd in a secure data center about an hour away that serves many Terminal Services remote apps to our local clients. We have a speedy network connection between our office and the data center and average about 99.9% uptime (during normal business hours) per year. However, there's always the opportunity for the internet to go down and thus all of our remote apps will cease to function.

Also, the remote apps must run from the data center (assume that there is a very very good reason for this that I cannot break) so placing a local copy is not possible.

Unfortunately our office is poorly wired: we only have a single "last mile" provider in town nor is it feasible to run new fiber to the building.

So, I need some short-term recovery techniques to get us thru a minor outage (lasting at most a single day). Anything more than a day and we'll go to a pre-designated off-site facility to continue operation.

We do have Cisco Client VPN capability so any IP-based network should suffice. I'd like to keep recurring maintenance costs to a minimum since it would be seldom used.

We have at most 15 people using Terminal Services and then about another 20-30 using the internet. The core requirement would be to keep those 15 people going; the rest of the office can continue to function (albeit at reduced output) without internet but if this solution can cover them that's a bonus.

I have a couple of different options that I can think of:

1. Cable modem. It's not particularly fast but it's fairly cheap to maintain on a monthly basis. Would be easy to go into the networking closet and swap out the internet to our gateway router. I don't know whether or not they'd piggy back on the same fiber to our location, though.
2. Wireless LAN. Something that uses the 3G wi-fi in the area. This seems like the most convenient but are there plans out there that don't cost $1M a month to maintain (something very cheap and then a reasonable bump up when we used it would be best) and require extensive hardware costs?
3. Satellite. Is that even fast enough to use RDP?
4. Consumer individual Wifi. What about AT&T (etc.) USB wifi cards? They are certainly fast enough to run a single user's RDP; the cards themselves are pretty cheap but the networking plans add up $$ really quick..

Is there another option I'm overlooking? What do you have for primary internet connectivity loss recovery (provided another last-mile provider is out of the question)? Thanks!!

Answer 1

A point to point T-1 to your colo would give you enough bandwidth to your terminal users, and you could then route your Internet users out the Internet connection at the colo.

Answer 2

We use a business cable connection as our secondary connection where I work. It's 16mbit down and 2mbit up. Since RDP traffic is mostly from the server to the client that should run a fair number of RDP sessions.

Also I know that cable companies are starting to roll out 50mbit service in some places so I would expect to see some faster options from your cable company at some point.

Answer 3

There are really only 2 options here with what you've laid out.

1. (Cheap) Use 3G/Cellular WAN. Check out the Kyocera KR-2 EVDO router. You could, for minimal cost, use this as a backup WAN connection for your network. For this to work, I would have to assume that your business application isn't doing any "complex displays" for Remote Desktop (i.e. just basic data entry / dialog boxes, etc., no animation or whatever)
2. (Expensive) Fundamentally the only way you can truly fix this problem is to bring the datacenter to you. I know your business application has a really good reason for being in the datacenter, but is it at all possible to establish a terminal services server in your local network synchronized with the datacenter? (with file sync or SQL mirroring or whatever).

Other than that, there's really no way to provide quality connectivity at high uptime when you have no options.

Answer 4

With any of the above solutions, I would talk to both your primary and secondary providers and discuss how they are connected to the secure data center. You might find out that they are routing through the same point of failure that you are using and that your redundancy is not actually as redundant as you think. So, you might be spending a chunk of money on redundancy and still have a single point of failure.

Also, with any wired solution, be sure that your primary and secondary cables aren't running in the same trench/path. Someone doing work could snap both cables at once.

Answer 5

Look around, there may well be a point to point wireless vendor out there, I know people who are very happy with that, for personal use they are paying $99 for a 6mbps symmetrical connection, and are very happy.

You might want to mention what part of the country you are in.

Answer 6

Approach 1

If you really only have a single Provider in town, there's no real technical way to get a backup, after all even the fallback location (assuming it's in the same town) depends on the same uplink provider. There could be another choice, that is swapping fallback location and main location, but I'd only consider that if the current fallback office geographically nearer to the datacenter than the current main office.

Consider what happens if some construction workers cut an important cable from your ISP. Going to the fallback location just to find the cut cable also breaks internet there? You won't see a happy boss if that happens and both locations are down.

I'd set up a proper support contract to make sure that you are a first priority customer.

Approach 2a

Contact your datacenter and ask them what the connections are they have, very often they will not only have large backbone connections but you might find a second ISP for a location near you (given you are already somewhere near the datacenter)

Approach 2b Aside from that I can't quite believe that there's no second provider in town, if not wired you could (depending on how much your downtime costs you) try to contact a network provider that goes with a satellite link.

As others have pointed out, a better long term solution might be to get an office in the datacenter so that you have a direct connection

Approach 3 (unlikely)

Swap datacenters :)

If moving to the datacenter is not an option, have the datacenter move to you. That is make a few calls to see if there's any viable option near you.

Answer 7

Satellite RTT will be to high to use RDP with good condition. I would choose Cable or 3G (but it's not as reliable as cable).

Answer 8

I know of a few Internet-enabled cafes near to the office, and I use a laptop. Zero monthly cost, and the cost of a couple of cups of coffee when I do need to use my alternate connection.