1

On a gluster file system with arbiter (quorum) volume, df report free/used space of arbiter volume insted of actual space available.

Scenario is made up of 3 server, one for quorum briks

For all servers:

[root@arbiter ~]# cat /etc/redhat-release CentOS Linux release
7.2.1511 (Core)
[root@arbiter ~]# glusterfs --version
glusterfs 3.7.15 built on Aug 30 2016 15:25:13

Peers

[root@arbiter ~]# gluster peer status
Number of Peers: 2

Hostname: node1
Uuid: ********************
State: Peer in Cluster (Connected)

Hostname: node2
Uuid: ********************
State: Peer in Cluster (Connected)

Brick size:

[root@arbiter ~]# df -hT /bricks/brick?
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdc1      xfs  1020M   43M  978M   5% /bricks/brick1
/dev/sdd1      xfs  1020M   34M  986M   4% /bricks/brick2

[root@node1,2 ~]# df -hT /bricks/brick?
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdc1      xfs   8.0G  493M  7.6G   7% /bricks/brick1
/dev/sdd1      xfs   8.0G   38M  8.0G   1% /bricks/brick2

Volume

[root@arbiter ~]# gluster volume info

Volume Name: glustervol1
Type: Distributed-Replicate
Volume ID: ********************
Status: Started
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: node1:/bricks/brick1/brick
Brick2: node2:/bricks/brick1/brick
Brick3: arbiter:/bricks/brick1/brick (arbiter)
Brick4: node1:/bricks/brick2/brick
Brick5: node2:/bricks/brick2/brick
Brick6: arbiter:/bricks/brick2/brick (arbiter)
Options Reconfigured:
performance.readdir-ahead: on

This fille system si mounted natively on a client as

[root@glusterclient ~]# mount -t glusterfs <anynode>:/glustervol1 /mnt/glustervol1

I expect about 8G of space available but df reports a disk size equal to arbiter volume:

[root@glusterclient ~]# df -hT /mnt/glustervol1
Filesystem           Type            Size  Used Avail Use% Mounted on
arbiter:/glustervol1 fuse.glusterfs  2.0G   77M  2.0G   4% /mnt/glustervol1

When it comes to disk usage the space seen by du is the space taken by files:

[root@glusterclient ~]# du /mnt/glustervol1 -h --max-depth=1
0       /mnt/glustervol1/.trashcan
258M    /mnt/glustervol1/test
3.4M    /mnt/glustervol1/test1
261M    /mnt/glustervol1

But I can't figure out how to check volume space actually available and taken; tried to googling for the answer but apparently no other person have the same use-case (nodes + arbiter)

Alessandro Carini
  • 312
  • 2
  • 13

1 Answers1

3

Glusterfs will only allow a volume to be the size of the smallest brick, even when using arbiter volumes. Your arbiter brick size needs to match the size of your other bricks.

This is clearly a job for LVM thin provisioning, as creating a huge thickly provisioned arbiter brick would be a foolish waste of space. This is yet another reason why this configuration is recommended, but it's rarely touched on.

Spooler
  • 7,046
  • 18
  • 29
  • In essence just: `\@`… – poige Dec 13 '18 at 15:24
  • 1
    I'm adding also another fact that comes as a surprise (in addition to finding out that GlusterFS's documentation is awfully incomplete): arbiter node would receive same traffic stream as typical node would, it only won't be stored. But bandwidth would be taken up by factor of all replicas, including arbiter. – poige Dec 13 '18 at 17:38