Inode usage gone up to 100% hence can't uninstall any package

Question

My server's inode usage has gone up to 100% due to which I am not able to uninstall extra linux-kernel-headers and linux-kernel-images

$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 523970    318  100% /
none           481856      2 481854    1% /sys/fs/cgroup
udev           480561    396 480165    1% /dev
tmpfs          481856    317 481539    1% /run
none           481856      1 481855    1% /run/lock
none           481856      1 481855    1% /run/shm
none           481856      3 481853    1% /run/user
/dev/xvdb      262144     11 262133    1% /mnt

Also HDD usage are up to only 78%

$ sudo df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  5.7G  1.7G  78% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            1.9G   12K  1.9G   1% /dev
tmpfs           377M  368K  377M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.9G     0  1.9G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdb       3.9G  8.1M  3.7G   1% /mnt

When I try to remove headers and images using sudo apt-get autoremove command, It gives me below error.

$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 linux-headers-3.13.0-83-generic : Depends: linux-headers-3.13.0-83 but it is not installed
E: Unmet dependencies. Try using -f.

When I do sudo apt-get -f install, It gives me below error.

dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb (--unpack):
 unable to create `/usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h'): No space left on device
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

When I do $ sudo dpkg --configure -a, It gives me below error.

    libpostfix-dns.so.1 -> libpostfix-dns.so.1.0.1
/sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~: No space left on device
dpkg: error processing package libc-bin (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: error: unable to create new file '/var/lib/dpkg/status-new': No space left on device

Now I am not sure how to tackle this issue. Any experts advice would be a great input. I am not sure from where should I start deleting files. Here is description of filesystem.

/$ sudo du -sh *
9.6M    bin
418M    boot
12K dev
6.9M    etc
216K    home
0   initrd.img
0   initrd.img.old
789M    lib
4.0K    lib64
16K lost+found
4.0K    media
28K mnt
684M    opt
du: cannot access ‘proc/21817/task/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/task/21817/fdinfo/4’: No such file or directory
du: cannot access ‘proc/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/fdinfo/4’: No such file or directory
0   proc
23M root
372K    run
9.4M    sbin
4.0K    srv
0   sys
19M tmp
2.9G    usr
669M    var
0   vmlinuz
0   vmlinuz.old

Note: I have already deleted log files older than 20 days. It didn't help to minimise inode usage and I am stuck now.

Shailesh Sutar · Answer 1 · 2016-11-15T13:19:21.307

In my case the / partition was showing as 100% inode usage due to which I was not able to uninstall any extra linux-headers and extra kernel linux-images. 1. First I deleted all the logs which were older than 20 days 2. I moved some files from /var/cache/ folder to /mnt/test/ folder which caused 3% free inodes on my system.

    $ sudo apt-get clean 
    $ sudo apt-get autoremove
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following packages will be REMOVED:
      linux-headers-3.13.0-67 linux-headers-3.13.0-67-generic
      linux-headers-3.13.0-68 linux-headers-3.13.0-68-generic
      linux-headers-3.13.0-70 linux-headers-3.13.0-70-generic
      linux-headers-3.13.0-71 linux-headers-3.13.0-71-generic
      linux-headers-3.13.0-73 linux-headers-3.13.0-73-generic
      linux-headers-3.13.0-74 linux-headers-3.13.0-74-generic
      linux-headers-3.13.0-76 linux-headers-3.13.0-76-generic
      linux-headers-3.13.0-77 linux-headers-3.13.0-77-generic
      linux-image-3.13.0-67-generic linux-image-3.13.0-68-generic
      linux-image-3.13.0-70-generic linux-image-3.13.0-71-generic
      linux-image-3.13.0-73-generic linux-image-3.13.0-74-generic
      linux-image-3.13.0-76-generic linux-image-3.13.0-77-generic
    0 upgraded, 0 newly installed, 24 to remove and 144 not upgraded.
    After this operation, 955 MB disk space will be freed.
    Do you want to continue? [Y/n] Y

After uninstalling those headers and images, inode usages are dropped to 64% now

$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 331671 192617   64% /
none           481856      2 481854    1% /sys/fs/cgroup
udev           480561    396 480165    1% /dev
tmpfs          481856    317 481539    1% /run
none           481856      1 481855    1% /run/lock
none           481856      1 481855    1% /run/shm
none           481856      3 481853    1% /run/user
/dev/xvdb      262144   1933 260211    1% /mnt

of course this is not the only way to deal with 100% inode usage problem. But In my case it worked since my /mnt/ partition had free inodes.

score 0 · Answer 2 · answered Nov 11 '16 at 18:05

0

Given your df -i output and seeing that you are not using any LVM - the only solution you have is to start deleting files.
Every file present on the system / partition will consume one inode.
Directories also consume inode.
So you need to figure out what's driving this number of files up.
Perhaps it's some process creating many temp files.
Perhaps it's log files.

You can use below command to see which directory is the biggest offender

find / -printf "%h\n"|grep -v "^/proc"|grep -v "^/sys"|cut -d\/ -f1-3|sort|uniq -c|sort -rn

answered Nov 11 '16 at 18:05

Dmitry Zayats

1,378
7
7

This doesn't work in my case since the inode is 100% it keeps giving me error as `no space left on device` see below extract. `sudo bash -c 'find / -printf "%h\n"|grep -v "^/proc"|grep -v "^/sys"|cut -d\/ -f1-3|sort|uniq -c|sort -rn'` `sort: cannot create temporary file in ‘/tmp’: No space left on device` `sudo find / -printf "%h\n"|grep -v "^/proc"|grep -v "^/sys"|cut -d\/ -f1-3|sort|uniq -c|sort -rn` `sort: cannot create temporary file in ‘/tmp’: No space left on device` – Shailesh Sutar Nov 11 '16 at 18:29
which puts me back to square one. – Shailesh Sutar Nov 11 '16 at 18:46
You're going to have to do it wtihout `sort` then, or you're going to have to find some empty directory somewhere on your computer and `mount -t tmpfs none /some/dir` then use `sort -T /some/dir` to use that tmpfs as the temporary directory. – DerfK Nov 11 '16 at 19:01
see if you can do this. Change to every dir from / and run ls -R1|wc -l. Perhaps that will be able to run and won't require temp file. You can also start moving your old logs to /mnt/. That should free some space. What are you running on this server which has potential to generate many log or temp files? – Dmitry Zayats Nov 11 '16 at 19:02
@DmitryZayats Logs are consuming only up to 2.3 MB. This is server from the cluster of zookeeper in production. Basically used for global locking during database migration – Shailesh Sutar Nov 11 '16 at 23:24
@DerfK Can you elaborate what you're trying to say here. – Shailesh Sutar Nov 11 '16 at 23:25
1

Size is not your problem, number of inodes is. Delete a few files before running this find command. The package manager cache is a likely target. You mentioned Ubuntu, so `apt-get clean` – John Mahowald Nov 12 '16 at 17:14

Inode usage gone up to 100% hence can't uninstall any package

2 Answers2