rsync : write failed - No space left on device (28)

Question

I want to rsync a folder from one server to another.

But rsync synchronization fails:

$ rsync -zr --compress-level=9 --delete /var/www/mywebsite/current/web/js login@192.168.1.4:/srv/data2_http
rsync: write failed on "/srv/data2_http/js/8814c77.js": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
rsync: connection unexpectedly closed (21747 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

But I have enough space !

$ du -h /var/www/mywebsite/current/web/js
2.4M    /var/www/mywebsite/current/web/js


df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          5.0G  3.0G  1.8G  64% /
/dev/root       5.0G  3.0G  1.8G  64% /
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           395M  132K  395M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            24K   16K  8.0K  67% /var/gandi
tmpfs            24K   16K  8.0K  67% /var/gandi
tmpfs           789M     0  789M   0% /run/shm
/dev/xvdb       202G  168G   25G  88% /srv/data2_http

df -i
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
rootfs           327680   67666   260014   21% /
/dev/root        327680   67666   260014   21% /
devtmpfs         504593     319   504274    1% /dev
tmpfs            504848     224   504624    1% /run
tmpfs            504848       2   504846    1% /run/lock
tmpfs            504848       5   504843    1% /var/gandi
tmpfs            504848       5   504843    1% /var/gandi
tmpfs            504848       2   504846    1% /run/shm
/dev/xvdb      13434880 2152940 11281940   17% /srv/data2_http

Can you manually create files on that file system? Sometimes when I have seen this situation, file system was somehow corrupted and needed fsck, or the hard drive was dying. — Janne Pikkarainen, May 05 '15 at 08:56
/var/www/mywebsite/current/web/js is the source. Check space on the target, 192.168.1.4:/srv/data2_http — sivann, May 05 '15 at 09:30
ext4. I'm doing a backup of my server, and I'll check after with fsck — Bouffe, May 05 '15 at 09:39
Does the user "login" have full write permission on the target directory? — Thorbjørn Ravn Andersen, May 05 '15 at 14:12

score 18 · Answer 1 · edited Jul 12 '17 at 20:05

I had the same issue, the destination directory had enough space but I'd get "No space left on device". Turns out rsync copies the file to somewhere else first, then moves it to the destination directory. To change this behavior, use --inplace.

According to https://download.samba.org/pub/rsync/rsync.html "This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file."

 rsync --inplace source destination

score 10 · Answer 2 · answered May 05 '15 at 09:51

10

rsync transfers contents to a temporary file, in the target folder, first; if it succeeds it renames that temporary file to become the target file. if the transfer fails, it deletes the temporary file. a 2GB file would have filled up your target space then after rsync deletes it the space is available again. so, rsync can trick your investigation of the space issue.

answered May 05 '15 at 09:51

Skaperen

1,094
2
11
23

I see this problem when trying to copy 200GB device with +500000 files to a 2TB NVMe device with >1TB free. So it is not really a mater of space. Something more weird can happen... Maybe a hardware issue? – Luis A. Florit Jun 25 '23 at 17:08

score 4 · Answer 3 · answered May 05 '15 at 10:10

4

1) Check destination space.

ssh login@192.168.1.4 
df -h /srv/data2_http

2) try the --inplace option of rsync. It prevents using more space, but makes destination files inconsistent during transfer.

answered May 05 '15 at 10:10

sivann

563
5
16

*--inplace** is what did the trick for me. I suspect the temporary file was created inside the */tmp* directory, the only one close to filling I have. The way I read it is that the temporary copy of the file, as discussed by *Skaperen* above, at least in my Debian, is created in */tmp*, which is the only rational explanation for the overflow I could find (like I said, **all** other filesystems are nearly empty!). And in fact, this also explains why the option **--inplace** solved my problem. But +1 for both of you guys, and a sincere thanks. – MariusMatutiae Oct 31 '16 at 16:11

score 0 · Answer 4 · edited Jun 30 '20 at 12:14

0

We called our hosting provider, we did some tests while the server is up. But nothing.

Finally, I just rebooted the server and it works now. I did not do this before because it's a production server

edited Jun 30 '20 at 12:14

kenlukas

3,101
2
16
26

answered May 06 '15 at 10:04

Bouffe

283
1
3
10

4

this should not be marked the correct answer. You did not have to reboot the server. – pedram bashiri Aug 24 '19 at 21:35

rsync : write failed - No space left on device (28)

4 Answers4