service apache2 restart leads to "Error: No space left on device"

Question

If I do

me@server:/home/me# service apache2 restart

on my LXC-VPS (node-server is not under heavy load) I'm getting

Error: No space left on device

df -h tells me that there is plenty enough space

/dev/loop1      394G     72G  303G   20% /
none            100K       0  100K    0% /dev
cgroup           12K       0   12K    0% /sys/fs/cgroup
tmpfs            63G       0   63G    0% /sys/fs/cgroup/cgmanager
tmpfs            63G       0   63G    0% /dev/shm
tmpfs            63G    903M   63G    2% /run
tmpfs           5,0M       0  5,0M    0% /run/lock
tmpfs            63G    9,0M   63G    1% /tmp
none             63G       0   63G    0% /run/shm

df -i tells me

Dateisystem      Inodes IBenutzt    IFrei IUse% Eingehängt auf
/dev/loop1     26214400  1947464 24266936    8% /
none           16488053       22 16488031    1% /dev
cgroup         16488053       16 16488037    1% /sys/fs/cgroup
tmpfs          16488053       17 16488036    1% /sys/fs/cgroup/cgmanager
tmpfs          16488053        1 16488052    1% /dev/shm
tmpfs          16488053      269 16487784    1% /run
tmpfs          16488053     1069 16486984    1% /run/lock
tmpfs          16488053      192 16487861    1% /tmp
none           16488053        1 16488052    1% /run/shm

This is the output of ipcs

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x4f3f00ec 9928704    root       600        1000       3                       
0x640109b7 9961473    root       600        1200712    3                       
0x0004e7b0 9437186    root       666        20564      0                       

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x002fa327 0          root       666        2         
0x00000000 9535489    www-data   600        1         
0x00000000 9568258    www-data   600        1         
0x7f9640a9 4096003    ftp_ase    644        3         
0x3780350e 4456452    ftp_sur    644        3

Any idea how to avoid this error?

@MyFault , please don't use comments to add information to the question. You can edit the text directly into your question (I've done it for you this time), and it works better for everyone. — MadHatter, Nov 23 '15 at 09:57
The host is running on a LXC container but the node system is not under heavy load. There are only 3 VPS with small LXC-containers. — MyFault, Nov 23 '15 at 10:14

score 3 · Answer 1 · answered Nov 23 '15 at 10:30

3

You can see this where apache is using IPC semaphores, probably from a apache module.

If it crashes it wont clean up the semaphore which remains permanently assigned to the system at least until a reboot.

Run ipcs and you'll probably see a lot of stuff owned by apache in there.

You can use the ipcrm command to clear it up.

answered Nov 23 '15 at 10:30

Matthew Ife

23,357
3
55
72

Hi @Matthew: I posted output of ipcs in my question above. I'm not sure if this is "a lot" or not. ;-) – MyFault Nov 23 '15 at 10:38
Ask your provider if you are sharing an IPC namespace with the container. – Matthew Ife Nov 23 '15 at 16:37
This did not solve it for me. I did `ipcrm --all`, but it still gives me the same error. – Gui Imamura Dec 26 '15 at 04:58

score 0 · Answer 2 · answered Nov 23 '15 at 09:06

0

You probably have a lot of small files and ran out of inodes (the structures that store a file metadata). If this is the case, you need to delete these files to free up space.

You can check the inode use via the command df -i.

answered Nov 23 '15 at 09:06

dr_

1,085
12
19

df -i shows the following: d.pr/1h4FK – MyFault Nov 23 '15 at 09:09

score 0 · Answer 3 · answered Nov 23 '15 at 09:53

0

Not sure about the error but it seems you are not using root user to restart service. Have you tried restarting apache2 with root user? What is the result?

answered Nov 23 '15 at 09:53

Kiran More

9
5

Hello @Kiran More: If I use root to restart the error is the same. – MyFault Nov 23 '15 at 10:13

service apache2 restart leads to "Error: No space left on device"

3 Answers3