0

Right now I have two glusterfs volumes

Volume Name: gv0
Type: Replicate
Volume ID: id-here
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: s1.example.com:/data/brick1/gv0
Brick2: s2.example.com:/data/brick1/gv0
Options Reconfigured:
performance.readdir-ahead: on

/etc/fstab (server1):

/dev/vdb1 /data/brick1 xfs defaults 1 2
s1.example.com:/gv0 /mnt/glusterfs glusterfs defaults,_netdev,direct-io-mode=disable 0 0

The glusterfs volume is mounted locally, yet read/writes are slow/sluggish. I know the connection between server1 and server2 is slow, but ideally it should write to the local volume, and then sync correct? I'm having issues where my upload application timesout when storing files to the locally mounted glusterfs volume.

I'm using the native FUSE client. All servers are on a KVM VM. Qcow2, no cache XFS filesystem for glusterfs partition.

Benchmarks

GlusterFS Vol:

[~]@s1:$ dd if=/dev/zero of=/mnt/glusterfs/zero1 bs=64k count=40  
40+0 records in
40+0 records out
2621440 bytes (2.6 MB) copied, 17.3101 s, 151 kB/s

Normal Vol

[~]@s1:~$ dd if=/dev/zero of=zero1 bs=64k count=40
40+0 records in
40+0 records out
2621440 bytes (2.6 MB) copied, 0.00406856 s, 644 MB/s
Dubb
  • 19
  • 4

1 Answers1

2

Writes on replicated volumes are always synchronized.

Some time ago, I specifically asked that on GlusterFS mailing list and the short reply was that it is not possible to have a background, continuous sync process while leaving the localhost as fast as possible.

A possible suggested workaround was to deliberately break the replication, write to your localhost, and restore the replication. The self healing daemon would kick in and background-sync all the changes.

This workaround clearly only works when your remote copy is read-only; if it is read-write, you will incur in split-brain scenarios. Anyway, if you only need a read-only remote copy, you can use GlusterFS's georeplication feature, which is based on rsync and it is fully decoupled from localhost writes.

shodanshok
  • 47,711
  • 7
  • 111
  • 180
  • That's unfortunate. Are there any viable alternatives? The servers are across data centers, so it eluded need to sync after write. I assume DRBD would have the same issue. I see a rsync watch script called syncd, but it seems difficult to scale past two servers. – Dubb Aug 03 '15 at 14:00
  • Yes, DRBD (even with it commercial proxy module) would have the same problem. I evaluated lsync2, but it is not really meant for master/master scenario and I found some corner case where it lead to data loss. Unfortunately I don't know any good,open,fast solution for these class of problems, and this is the precise reason why *I am writing my own rsync-based sync software just now*, but it is not production ready yet. – shodanshok Aug 03 '15 at 14:18
  • Have you looked at BTSync? Seems to watch a directory and transfer files via the BitTorrent protocol. – Dubb Aug 03 '15 at 15:54
  • Yes, but I need to transfer POSIX ALCs and EA (extended attributes), and may tools simply did not support that (Unison, btsync, syncthing, ecc). – shodanshok Aug 03 '15 at 16:36
  • I see BTSync is closed source what about https://syncthing.net – Dubb Aug 03 '15 at 16:42
  • As stated above, it does not support ACLs nor EAs. As a side note, if you are satisfied with the reply, consider marking it as accepted :) – shodanshok Aug 03 '15 at 16:46
  • I posted that comment without refreshing. I'll check out the other tools you mentioned since i don't need that many features. Thanks – Dubb Aug 03 '15 at 16:55
  • What's wrong with georeplication as mentioned in the answer? – S19N Aug 07 '15 at 09:24
  • Georeplication is OK for a read only remote copy. For a remote read/write copy (with two way replication), georeplication is not appropriate. – shodanshok Aug 07 '15 at 13:02