rsync (and scp) failing with no space left on XFS device

Question

I have a fresh install of CentOS 7 with a primary partition of 400GB formatted for XFS. Less than one GB is currently in use. I am trying to rsync 65GB of files over and the copy starts failing between 14GB and 26GB. (I have tried multiple times.)

I have plenty of space:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      400G  919M  400G   1% /
devtmpfs        7.3G     0  7.3G   0% /dev
tmpfs           7.2G     0  7.2G   0% /dev/shm
tmpfs           7.2G   41M  7.2G   1% /run
tmpfs           7.2G     0  7.2G   0% /sys/fs/cgroup
/dev/xvdb        37G   49M   35G   1% /mnt

I have plenty of inodes:

# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       400M   24K  400M    1% /
devtmpfs         1.9M   289  1.9M    1% /dev
tmpfs            1.8M     1  1.8M    1% /dev/shm
tmpfs            1.8M   280  1.8M    1% /run
tmpfs            1.8M    13  1.8M    1% /sys/fs/cgroup
/dev/xvdb        2.4M    11  2.4M    1% /mnt

I don't know what this means, but I have seen it referenced in other places, so I thought it would be helpful in diagnosing:

# xfs_db -r "-c freesp -s" /dev/xvda1
   from      to extents  blocks    pct
      1       1      82      82   0.00
      2       3      14      30   0.00
      4       7      12      63   0.00
      8      15      13     131   0.00
     16      31      17     369   0.00
     32      63       9     408   0.00
     64     127       8     774   0.00
    128     255       9    1702   0.00
    256     511       4    1605   0.00
    512    1023       6    4267   0.00
   1024    2047       1    1664   0.00
   2048    4095       2    7446   0.01
   8192   16383       2   24221   0.02
  32768   65535       1   33389   0.03
 131072  262143       1  205592   0.20
 262144  524224     200 104345607  99.73
total free extents 381
total free blocks 104627350
average free extent size 274612

I have no idea what is left to check, or why it keeps failing. The quoted sections above are from before the copy. After the copy, inodes are at 47K.

Thank you in advance for any help you can provide.

Edit: Using CentOS 7, not CentOS 6.7. Also, the kernel version:

# uname -r
3.10.0-123.8.1.el7.x86_64

Edit 2:

Here are the results after another attempt (22GB/65GB completed before failure):

# touch test
touch: cannot touch ‘test’: No space left on device
# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      400G   22G  379G   6% /
devtmpfs        7.3G     0  7.3G   0% /dev
tmpfs           7.2G     0  7.2G   0% /dev/shm
tmpfs           7.2G   41M  7.2G   1% /run
tmpfs           7.2G     0  7.2G   0% /sys/fs/cgroup
/dev/xvdb        37G   49M   35G   1% /mnt
# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       400M   47K  400M    1% /
devtmpfs         1.9M   289  1.9M    1% /dev
tmpfs            1.8M     1  1.8M    1% /dev/shm
tmpfs            1.8M   280  1.8M    1% /run
tmpfs            1.8M    13  1.8M    1% /sys/fs/cgroup
/dev/xvdb        2.4M    11  2.4M    1% /mnt
# xfs_db -r "-c freesp -s" /dev/xvda1
   from      to extents  blocks    pct
      1       1      60      60   0.00
     64     127       1      81   0.00
    128     255       1     186   0.00
    512    1023       3    2317   0.00
   1024    2047       2    2538   0.00
   2048    4095       1    2310   0.00
   4096    8191       1    8001   0.01
   8192   16383       2   25782   0.03
 262144  524224     189 99059740  99.96
total free extents 260
total free blocks 99101015
average free extent size 381158

Even as root I am unable to create any new files or directories.

Edit 3: Additional requested information:

# xfs_info /dev/xvda1
meta-data=/dev/xvda1             isize=256    agcount=201, agsize=524224 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=4096   blocks=104855999, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

@Wesley Many repetitions of "rsync: recv_generator: mkdir "" failed: No space left on device (28)" or "rsync: mkstemp "" failed: No space left on device (28)" — BamaPookie, Dec 30 '15 at 19:32
Are you able to create files on the filesystem locally (not using scp or rsync)? If so, anything relevant in the journal or audit log? I have seen oddly worded errors come from SELinux preventing access on more than on occasion. — Jon Moore, Dec 30 '15 at 20:07
It looks like that `xfs_db` data isn't useful here, what could be helpful to figure out some of where the problem *might* be would be `xfs_info /dev/xvda1`, which will give us some details about how the xfs filesystem is set up. — sysadmin1138, Dec 30 '15 at 21:12
@JonMoore I tried as you suggested and was unable to mkdir or touch a new file. (See Edit 2 above). I see nothing related in messages or the audit log. — BamaPookie, Dec 30 '15 at 21:15
@sysadmin1138 I added the xfs_info to the original question under edit 3. — BamaPookie, Dec 30 '15 at 21:19

score 3 · Accepted Answer · answered Dec 30 '15 at 22:12

3

Given what you've posted, there are only a few more places in XFS that can cause this problem. Here are two:

Quotas. These aren't exactly easy to accidentally turn on, but it's another area worth checking if nothing else yields results. Look for a uquota mount-option. If you still have suspicions, noquota as an option will tell everything to stop quotaing.
Mount options. You should have inode64 in your mount options, which is the mount-flag to tell the kernel to use 64-bit inode allocation. If you don't have it, the kernel could throw ENOSPC (out of space) when attempting to create new files and directories. Your partition isn't big enough for this to be a thing, but we're out of easy things to check at this point.

There is one other thing to check, outside of XFS.

Open File-Handle Ulimit This might be what rsync is running into. If it was memory, you would be seeing a different error.

answered Dec 30 '15 at 22:12

sysadmin1138

133,124
18
176
300

It could be the ulimit. I will check that when I get back in front of my computer. – BamaPookie Dec 30 '15 at 22:19
The mount options do include inode64 and noquota (and rw, relatime, seclabel, and attr2 as well). – BamaPookie Dec 30 '15 at 23:52
`ulimit -Hn` was 4096. I increased it to 10240. `ulimit -Sn` was 1024. I increased it to 4096. – BamaPookie Dec 30 '15 at 23:59
It looks like it was the ulimit. After changing the ulimit and rebooting, I was able to complete the rsync. – BamaPookie Dec 31 '15 at 01:58

rsync (and scp) failing with no space left on XFS device

1 Answers1