58

I am getting the "No space left on device" error for pretty much anything i try to do. Even using tab to autocomplete a command!

but when i do df -h i get:

ubuntu@ip-10-0-2-108:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       99G  6.5G   88G   7% /
udev            3.7G  8.0K  3.7G   1% /dev
tmpfs           1.5G  184K  1.5G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdb       414G  199M  393G   1% /mnt
overflow        1.0M  1.0M     0 100% /tmp

which to me looks like there is tons of space. df -i also looks similar:

ubuntu@ip-10-0-2-108:~$ df -i
Filesystem       Inodes IUsed    IFree IUse% Mounted on
/dev/xvda1      6553600 94227  6459373    2% /
udev             951353   393   950960    1% /dev
tmpfs            953649   274   953375    1% /run
none             953649     3   953646    1% /run/lock
none             953649     1   953648    1% /run/shm
none             953649     1   953648    1% /run/user
/dev/xvdb      27525120    11 27525109    1% /mnt
overflow         953649    12   953637    1% /tmp

I am on an Amazon EC2 ubuntu 12.04 instance.

Here are some examples of the error popping up:

ubuntu@ip-10-0-2-108:~$ sudo crontab -e
/tmp/crontab.RvYjrR/crontab: No space left on device

ubuntu@ip-10-0-2-108:~$ ls /va (hit tab for autocomplete)

-bash: cannot create temp file for here-document: No space left on device
-bash: cannot create temp file for here-document: No space left on device

however the server seems to be running and everything seems to be working. what on earth is going on??

Or Gal
  • 1,298
  • 3
  • 12
  • 20
  • 14
    Did you see that you `/tmp` is 100% full. ? – iamauser Sep 09 '13 at 20:27
  • 4
    This question appears to be off-topic because it is about Ubuntu; should be on Unix or Askubuntu – Keith Thompson Sep 09 '13 at 20:46
  • 7
    For those that came here looking for answers but still couldn't figure out what was going on, check your Inodes. Run `df -i` and see if you `IUse%` is close or at 100. You should delete as many 0 byte or small files as you can spare if so. – sjagr Apr 29 '16 at 19:55
  • 2
    The fastest way to locate your folders that are too full is by narrowing down the folder file size in levels from the root folder. You start with the root folder by: `sudo du -h --max-depth=1 /` Then - EITHER you increase the depth, i.e. the levels below: `sudo du -h --max-depth=2 /` OR - quicker - you you look which folder has eaten up the most disk space, and do the same on this folder: `sudo du -h --max-depth=1 /home//` Once you found it, just remove that one: `rm -rf ` – Agile Bean Jun 10 '18 at 03:51
  • @AgileBean This was really helpful to seekout and find the log file of pm2 which had been the cause to full the disk : Thanks – Rakshitha Muranga Rodrigo Dec 20 '18 at 04:46
  • @RakshithaMurangaRodrigo glad it helped you! – Agile Bean Dec 20 '18 at 05:06

1 Answers1

31

It's possible that you've run out of memory or some space elsewhere and it prompted the system to mount an overflow filesystem, and for whatever reason, it's not going away.

Try unmounting the overflow partition:

umount /tmp

or 

umount overflow
Jon Lin
  • 142,182
  • 29
  • 220
  • 220
  • that works. had to kill tomcat first because it was keeping the device busy. looks like tomcats temp dir was filling up or something. – Or Gal Sep 10 '13 at 07:21