How can one file `/etc/hosts` with 256 bytes use `1.9M` inode?

Question

I use docker to do something. But the inode was exhausted after running about 15 days. The output of df -i in docker was:

Filesystem       Inodes   IUsed    IFree IUse% Mounted on
overlay         3276800 1965849  1310951   60% /
tmpfs          16428916      17 16428899    1% /dev
tmpfs          16428916      15 16428901    1% /sys/fs/cgroup
shm            16428916       1 16428915    1% /dev/shm
/dev/vda1       3276800 1965849  1310951   60% /etc/hosts
tmpfs          16428916       1 16428915    1% /proc/acpi
tmpfs          16428916       1 16428915    1% /proc/scsi
tmpfs          16428916       1 16428915    1% /sys/firmware

The hosts file content:

127.0.0.1 xxx xxx
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4

::1 xxx xxx
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

Why the hosts can use so much inodes? How to recover it?

Have you done an `fsck` on the dev file? I haven't used any form of `*nix` in a long time but there should be an analysis switch that doesn't make changes. Check the man page. Then you can repair the disk when you feel confident it's the right thing to try. — WJS, Mar 09 '21 at 02:15
what happens if you run `docker volume ls` from the host? Also try running `docker volume prune` (again, on the host). — Z4-tier, Mar 09 '21 at 02:45

BMitch · Answer 1 · 2021-03-22T11:43:52.017

Inside the container, that file is a bind mount. The mount statistics come from the underlying filesystem where the file is originally located, in this case /dev/vda1. They are not the statistics for the single file, it's just the way mount shows this data for a bind mount. Same happens for the overlay filesystem since it's also based on a different underlying filesystem. Since that filesystem is the same for each, you see the exact same mount statistics for each.

Therefore you are exhausting the inodes on your host filesystem, likely the /var/lib/docker filesystem, which if you have not configured a separate mount, will be the / (root) filesystem. Why you are using so many inodes on that filesystem is going to require debugging on your side to see what is creating so many files. Often you'll want to separate docker from the root filesystem by making /var/lib/docker a separate partition, or symlinking it to another drive where you have more space.

As another example to show that these are all the same:

$ df -i /var/lib/docker/.
Filesystem                          Inodes   IUsed    IFree IUse% Mounted on
/dev/mapper/bmitch--t490--vg-home 57098240 3697772 53400468    7% /home

$ docker run -it --rm busybox df -i
Filesystem              Inodes      Used Available Use% Mounted on
overlay               57098240   3697814  53400426   6% /
tmpfs                  4085684        17   4085667   0% /dev
tmpfs                  4085684        16   4085668   0% /sys/fs/cgroup
shm                    4085684         1   4085683   0% /dev/shm
/dev/mapper/bmitch--t490--vg-home
                      57098240   3697814  53400426   6% /etc/resolv.conf
/dev/mapper/bmitch--t490--vg-home
                      57098240   3697814  53400426   6% /etc/hostname
/dev/mapper/bmitch--t490--vg-home
                      57098240   3697814  53400426   6% /etc/hosts
tmpfs                  4085684         1   4085683   0% /proc/asound
tmpfs                  4085684         1   4085683   0% /proc/acpi
tmpfs                  4085684        17   4085667   0% /proc/kcore
tmpfs                  4085684        17   4085667   0% /proc/keys
tmpfs                  4085684        17   4085667   0% /proc/timer_list
tmpfs                  4085684        17   4085667   0% /proc/sched_debug
tmpfs                  4085684         1   4085683   0% /sys/firmware

From there you can see /etc/resolv.conf, /etc/hostname, and /etc/hosts are each bind mounts going back to the /var/lib/docker filesystem because docker creates and maintains these for each container.

If removing the container frees up a large number of inodes, then check your container to see if you are modifying/creating files in the container filesystem. These will all be deleted as part of the container removal. You can see currently created files (which won't capture files created and then deleted but still held open by a process) with: docker diff $container_id

But when I use `docker stop xxx` and `docker rm xxx`, it's freed. And so on, I use `docker run -v xxx:xxx` to mount the worker directory (not /etc/hosts) in docker to host filesystem, it's not increasing any more. — herbertluo, Mar 10 '21 at 04:07

How can one file `/etc/hosts` with 256 bytes use `1.9M` inode?

1 Answers1