The real question is how do I provide two routes to the Internet and ensure my hosts always use the fastest path to get there?
The case for routing. You have two locations, each with its own Internet connection. Hosts at each location are best served by using their site's route to the Internet. The two locations are also connected by a single link. This path allows hosts in site A talk to hosts in site B and vice-versa. It can also be leveraged to provide each site with an alternate route to the Internet (lucky you).
The proper way to configure this network is to put a router at each site, connect each site's Internet connection to its respective router, and then connect the two routers via the existing site-to-site link.
How it works. Each site needs its own subnetwork address, but each site's hosts will always use their location's route to the Internet, and when a site's Internet connection fails, the routers do their job and begin routing traffic to the other site's router which in turn forwards it to the Internet.
Is this worth all the trouble?
Yes, for several reasons:
- Faster Internet. You'll take advantage of the aggregate bandwidth of both of your Internet connections, not just one.
- Faster site-to-site link. Your location-to-location link won't be used by one site's hosts for Internet traffic, freeing up the bandwidth for other uses.
- Less failover delay. If a host is configured with two gateways and the one in use goes down, the client will experience a delay before trying the other gateway. The potential for this to continue exists as long as the gateway is down. Moving the failover responsibility to the router consolidates this delay into a single point in time, and once it's done, it's done.
- Host compatibility. Some hosts won't let you configure multiple gateways. If the one they're using fails, that host can't access the Internet until the gateway comes back up or you re-configure the host.
- No double work. You'll end up using routers anyway if you have problems assigning hosts two gateways.