1

We have two offices, one in London and one in Shanghai, each needing to be able to access the same set of files. This means we need a solid, speedy means of synchronising a set of folders between servers at either office. They're likely to be Windows servers, but we could look at Linux boxes if the software side makes more sense on *nix.

We've considered Rsync, Unison, Gluster, and a few other options, but none of them seem capable of reliably keeping the servers in sync between such distant office locations. Each office is on DSL connectivity over the open internet, so encryption is also a factor.

Does anyone have any hints for getting the servers synchronising in as close to real time as possible, without dying constantly?

Andy

Mike Pennington
  • 8,305
  • 9
  • 44
  • 87
Andy S
  • 11
  • 1
  • Fundamentally: you're boned. It only takes one 50MB PPT and your DSL line is clogged for minutes. Also, concurrent-edit conflict resolution and a host of other practical considerations put the kybosh on this good and hard. – womble Jun 28 '12 at 14:38
  • Dropbox, sounds like an easy, and cheap/free solution depending on the amount of data you use. – DanBig Jun 28 '12 at 15:29

2 Answers2

4

Since they're Windows servers, have you considered DFS?

Whether it's acceptable or not will probably depend on what you mean, precisely, when you say "as close to real time as possible" (and some clarity on that might help in getting a better answer to your question), as well as what the specific use case here is (amount of data and frequency of change), but it's probably a good starting point.

It would also help to know what the speed of your DSL connections are, because that would factor into whether you need to look at a protocol with native compression too (have you tried rsync -z to add compression?), but bottom line is that if you need to move more data than your pipes can handle, you either need bigger pipes, or need to limit yourself to synchronizing less data.

HopelessN00b
  • 53,795
  • 33
  • 135
  • 209
  • 1
    You might wanna add that DFS-R _greatly_ enhances replication across long distance/slow links as long as the files are not deleted and re-created at one side. – pauska Jun 28 '12 at 14:57
3

If you have an IT budget, one option is a WAN optimizer, which is essentially a local network cache; keep in mind that these devices can be very specialized for what applications / protocols get cached. These devices also ensure that cached-copies are the latest version.

As an example, this is a deployment guide for Sharepoint services on Cisco's WAAS appliance.

There are many vendors in the market besides Cisco. If you google "WAN optimization", you'll get the major players.

Mike Pennington
  • 8,305
  • 9
  • 44
  • 87
  • Considering the comment "Each office is on DSL connectivity over the open internet, so encryption is also a factor" and the fact that we're talking about synchronizing file changes between two offices, would WAN optimizing appliances actually provide a noticeable benefit? I wouldn't think so, but my knowledge in that area's pretty weak, so I'm genuinely curious. – HopelessN00b Jun 28 '12 at 15:02
  • It depends on where encryption happens. Even if you encrypt with IPSec on a WAN router, you put the cache on the LAN of each site. Regardless of whether you use DFS or a cache engine, the bottom line is there is no magic... the diffs of the file have to be replicated. QoS can help manage these issues if you get a boggy WAN from replication – Mike Pennington Jun 28 '12 at 15:06