1

**** Hoster something changed on hypervizor but said, that nothing was changed. The problem is not on the server anymore... ****

enter image description hereCould someone help me to solve the following problem:

If I want to restart the daemon, I have the notification:

Error: No space left on device

lsof | grep deleted:

rewrite-0 11111 11153     nginx  mem       REG               0,96           598535772 (deleted)/dev/zero (stat: No such file or directory)
rewrite-0 11111 11153     nginx  mem       REG               0,96           598535778 (deleted)/dev/zero (stat: No such file or directory)
rewrite-0 11111 11153     nginx  mem       REG               0,96           598535776 (deleted)/dev/zero (stat: No such file or directory)

And a lot of other string for php-fpm, mysql..

This is VPS Centos. Can't find any solution for that. Would be very appreciate for the help!

#### UPDATED
Filesystem         Size  Used Avail Use% Mounted on
/dev/ploop16716p1  200G   11G  181G   6% /
devtmpfs           3.0G     0  3.0G   0% /dev
tmpfs              3.0G     0  3.0G   0% /dev/shm
tmpfs              3.0G  248K  3.0G   1% /run
tmpfs              3.0G     0  3.0G   0% /sys/fs/cgroup
tmpfs              615M     0  615M   0% /run/user/1000
tmpfs              615M     0  615M   0% /run/user/0

ls -la /dev/zero

crw-rw-rw- 1 root root 1, 5 Aug 21 16:21 /dev/zero

df -i

Filesystem          Inodes  IUsed    IFree IUse% Mounted on
/dev/ploop16716p1 13295616 353527 12942089    3% /
devtmpfs            786432     65   786367    1% /dev
tmpfs               786432      1   786431    1% /dev/shm
tmpfs               786432    221   786211    1% /run
tmpfs               786432     10   786422    1% /sys/fs/cgroup
tmpfs               786432      1   786431    1% /run/user/1000

[root@mail ~]# dd if=/dev/zero of=/file bs=1G count=50

50+0 records in
50+0 records out
53687091200 bytes (54 GB) copied, 168.672 s, 318 MB/s

[root@mail ~]# du -hs /file

51G     /file

[root@mail ~]# df -h

Filesystem         Size  Used Avail Use% Mounted on
/dev/ploop16716p1  200G   61G  131G  32% /
devtmpfs           3.0G     0  3.0G   0% /dev
tmpfs              3.0G     0  3.0G   0% /dev/shm
tmpfs              3.0G  252K  3.0G   1% /run
tmpfs              3.0G     0  3.0G   0% /sys/fs/cgroup
tmpfs              615M     0  615M   0% /run/user/1000
tmpfs              615M     0  615M   0% /run/user/0

For any reason:

repquota -a:

*** Report for user quotas on device /dev/ploop16716p1
Block grace time: 7days; Inode grace time: 7days
                        Block limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      -- 7523364       0       0         251948     0     0       
nobody    --      20       0       0              5     0     0       
smmsp     --      92       0       0             23     0     0       
rpc       --       4       0       0              2     0     0       
apache    --      16       0       0              4     0     0       
mysql     --  180824       0       0            384     0     0       
named     --      12       0       0              3     0     0       
user      --  236604       0       0           1081     0     0       
clamupdate --  260460       0       0              5     0     0       
dovecot   --       4       0       0              1     0     0       
postfix   --     728       0       0             80     0     0       
vmail     -- 1415436       0       0          14824     0     0       
spamd     --      32       0       0              9     0     0       
nginx     --  775704       0       0          57129     0     0       
amavis    --    4644       0       0             31     0     0       
postgrey  --   10340       0       0              9     0     0       
opendkim  --      40       0       0             10     0     0       
opendmarc --     212       0       0              3     0     0       
redis     --     284       0       0             16     0     0       
ossec     --   29108       0       0           1216     0     0       
ossecr    --     284       0       0              5     0     0       
ossecm    --       4       0       0              1     0     0       
#500      --   97932       0       0           5318     0     0       
#1001     --    7024       0       0            435     0     0       
#984      --     532       0       0            136     0     0       
#501      --   22356       0       0           2645     0     0 

I've contacted the hosting provider and they tell me, that the problem they can't find, but logs saying:

Aug 24 07:43:17 mail yum[17224]: Installed: strace-4.12-6.el7.x86_64
Aug 24 07:53:43 mail clamd[1484]: SelfCheck: Database status OK.
Aug 24 07:55:58 mail systemd-tty-ask-password-agent: Error: No space left on device
Aug 24 07:55:58 mail systemctl: Failed to stop systemd-ask-password-plymouth.path: Unit systemd-ask-password-plymouth.path not loaded.
Aug 24 07:55:58 mail systemctl: Failed to stop systemd-ask-password-plymouth.service: Unit systemd-ask-password-plymouth.service not loaded.

And after that restart for daemons is working without any problem. Any ideas, dear community?

ipcs -a

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      

------ Semaphore Arrays --------
key        semid      owner      perms      nsems  
kbu
  • 255
  • 4
  • 14
  • 1
    Do you actually have enough space on `/`? The `/dev/ploop....` indicates a loopback file system on the VPS host and if the hoster overcommited storage, the hosting file system might be full. – Sven Aug 23 '18 at 12:07
  • Check if you can create a file. In the situation I mean, you would get a report that you have plenty of space left, but that's your storage quota. The host FS might be full anyway. – Sven Aug 23 '18 at 12:11
  • [root@mail ~]# touch /testfiles [root@mail ~]# ls -la /testfiles -rw-r--r-- 1 root root 0 Aug 23 14:14 /testfiles – kbu Aug 23 '18 at 12:14
  • succesfully created – kbu Aug 23 '18 at 12:14
  • Also nothing in /var/log/messages or in journalctl – kbu Aug 23 '18 at 12:17
  • You can create empty files all day. But can you create files with data in them? This is almost 100% your VPS provider has run out of disk space. OpenVZ VPSes lie about how much disk space they really have available. – Michael Hampton Aug 23 '18 at 14:05
  • Updated the topic. I can create files without any problem. With dd created file 50G – kbu Aug 23 '18 at 14:24
  • 1
    OK, i re-read your question. But something important is missing: What gave you the error message? – Michael Hampton Aug 23 '18 at 15:16
  • When I do: systemctl restart "any-daemon". I have the error message: Error: No space left on device – kbu Aug 23 '18 at 20:08
  • Normally when a process has a deleted file open the kernel will present it with ` (deleted)` appended to the name. But in your case you have `(deleted)` in the beginning (without a space). I think that difference will provide a clue. Possibly it didn't actually open `/dev/zero` but rather a path ending with `/dev/zero` in which an intermediate directory has been deleted, unmounted, or is otherwise inaccessible. – kasperd Aug 23 '18 at 20:45
  • Updated the topic – kbu Aug 24 '18 at 06:27

2 Answers2

0

Since everything else people have suggested for you has failed, I'll try a more exotic one -- has your system for some reason run out of semaphores?

If ipcs -a shows you a long list, then it might be worthwhile to remove the most obvious ones (like, semaphores existing for a user which only should run some service which has already been stopped) with the ipcrm command.

Janne Pikkarainen
  • 31,852
  • 4
  • 58
  • 81
0

The problem soved for now with sysctl parameter: fs.inotify.max_user_watches = 10240

kbu
  • 255
  • 4
  • 14