WAN Design Questions

Question

I've been tasked with redesigning our company's WAN in North America. We have two offices in the U.S., one in NY and one in the midwest. We also have offices in Europe. I won't be touching anything in Europe, with the exception of joining the Euro WAN with the U.S. WAN.

The current topology is as follows:

We have an offsite hosting facility in PA where the WAN, globally, ties together. The internet connection for the two US sites comes in through here, via a PIX firewall. Internet connectivity to NY is fine, but to the midwest office there are latency issues. Each site connects to PA via a T1, and all traffic, both network and internet, is carried over these T1 links.

The future topology:

The off-site hosting facility will be done away with and will in effect be moved to our midwest location. This is where the tying together of the WAN will occur.

I'm looking for suggestions as to how to best design this in terms of speed, cost, and security. Currently, we only have one point of access to the internet in the US, which is via the off-site hosting facility. I'm thinking that for the sake of providing a fast, reliable internet connection to each office, it would be better to have individual connections to each site instead of having one net connection going to midwest, and then pulled over the WAN link to NY. Business class cable connection is what I had in mind, each with their own firewall of course. Then, connect the two offices together with a T1...or are there faster, equally reliable, and less expensive methods to connect the two together?

Please poke holes in my ideas, as I need to better understand what's available from a design perspective.

Answer 1

I second Evan Anderson's suggestion of MPLS. It's the solution that I have considered for my own network.

Logically, it's point to cloud, and all the complex routing is done by the provider. This necessitates that you have one unified provider across your entire infrastructure*. This isn't as bad as it might sound, since there's a bit of configuration collaboration to be done between you and your provider. I wouldn't want to do it multiple times.

Also, be aware that for not-much-more-than-a-T1, you can get into the metro-ethernet type solutions. This is excellent, even if you start out with something like 5Mb/s, because you can expand in the future without adding more lines. It might not be important in your branches, but it sounds like your "central" location could definitely use more than a T1. Take care to mention that to the providers you are shopping around with. Some of them don't (or at least didn't 2 years ago) mix their T1 and ethernet MPLS clouds. That may or may not be the case now.

If I were in your situation, I'd go with a very large provider where you will have the option of connecting your European nodes to the MPLS network as well. There aren't many (or at least there wasn't when I was looking).

-* There are 3rd party providers that can bridge the gap between carriers. Essentially they connect to both and manage your connection on both of them. Blargh.

Answer 2

Consider looking at MPLS offerings from various national providers (QWest, AT&T, etc). You can get Internet connectivity at each remote office and "private" connectivity between the sites across the provider's "cloud". Conceptually, it's a bit like having a VPN between the sites, except the provider's gear is handling the VPN connectivity.

You might find out that the monthly cost isn't as bad as you think. If you bring your voice telephone into the picture you may actually end up with a cost savings.

Edit (now that I have some more time):

Some "nice" things that MPLS providers will often give you "for free" with the solution include:

• NAT and firewalling inside the cloud. Can be a cost savings for bringing the MPLS into sites that don't already have firewalls, but typically isn't as flexible as having your own firewall on-site (and isn't as easy to audit or get counters out of). I've got a Customer who has an MPLS provider that lets us use their firewall functionality at the sites that don't host Internet-facing servers, but passes all traffic at the site where Internet-facing servers are hosted in to the firewall there with no filters or NAT. It's actually rather nice.

• QoS for traffic crossing the MPLS cloud. If you know you're going to have specific traffic flows between sites on the cloud that need to have priority you can usually have this provisioned in the cloud (and on the CPE at each site).

• Central egress to the Internet, if you want that. At the expense of using your cloud bandwidth you can typically request a configuration that causes all access to the Internet from the remote sites to traverse the cloud and egress at a "hub" site.

You can have some flexibility with using multiple ISPs. I have one Customer who is using MPLS for the "important" remote sites, and VPN connections using Cisco ASA devices and commodity DSL or cable Internet connections for the "less important" sites. (They are running a VPN concentator device at their "hub" site that is exposed through the Internet via the MPLS provider.)

In theory, you could also use a dynamic routing protocol or floating static routes to do "failover" to VPN tunnels across secondary commodity Internet connections if the MPLS cloud "went down". If your uptime needs justify the expense, look into that. The typical problems with MPLS networks happen in the "last mile" (typical boneheaded telco issues-- smart jacks failing, fibers being BIFF'ed, etc), though, not in "the cloud".

Answer 3

I think you're on the right track. Definitely only load up your VPN/branch office connections with actual branch office traffic. Leave the internet traffic up to the local provider individually at each site unless you have some higher-security or centralized filtering/auditing reasons to take it through the main datacenter.

Answer 4

You could also do IPSEC VPN Tunnels over the internet with the business class connections. If you get ones with a good upload speed, you will probably have more bandwidth. However, you will not have QoS since it goes over the internet and in theory the T1 might be more reliable.

You can also consider using an MPLS provider to connect all of your offices. The provider will be able to implement some QoS for you, and then you could use the separate connection for the internet. The provider will isolate your traffic so it acts as if you have a VPN between the two. Basically, with MPLS the provider does a lot more for you, which can be either a disadvantage or an advantage.

Answer 5

Several of the other answers implied this, but consider using the Internet connection at each site as a backup connection to either the other US location or the one in Europe. You can setup a site-to-site VPN that automatically (floating static or higher administrative cost) kicks in if your primary link goes down. This is especially good if they are through different carriers.

Answer 6

One question I have is what is the cost of this project and what are the goals? Are you trading stocks in real-time over this network or just keeping remote offices in touch with the head office for payroll and inventory? Is there any interest in running voice over this? Will someone ask you to run video over it a week after you buy all your equipment?

The cheapest way to get fast links to the internet is to go with DSL / fios / cable from local providers, get 2 independent links, then run VPNs over those and routing protocols over the VPNs so you have reliable failover over the "internal" network. This is also stupid complex and likely harder to keep operational in the long run, but cheap and fast.

If you're willing to write a big check, get MPLS from one provider and let them manage your equipment and network. But, is the same MPLS provider able to reach all of your office sites? If not, you'll have to run a VPN between them (MPLS providers don't typically offer MPLS tunnels between providers). If you've got a nation-wide operation, you'll be limited to 3-8 providers with reach into all your sites. If you're a global operation, that number got a lot smaller unless you're willing to bribe providers into talking to each other.

I would avoid T1s -- they are really slow and very expensive for what you get. I've also seen really poor reliability from some providers. Fractional T3 is often not a whole lot more, but often you can also just get ethernet from the same providers who offer the T1/T3s.