I am getting some very confusing results from the Telegraf Disk Input Plugin. It's telling me that a disk that's 19% used is at 100%. This experiment works fine on my laptop (Ubuntu) but on the server (Ubuntu) it's giving me erroneous data. I have installed Telegraf on the server locally (not using Docker) and it's actually correct.
I ran Telegraf in Docker with this command.
docker run -v /:/hostfs:ro -v $PWD/test.conf:/etc/telegraf/telegraf.conf:ro -e HOST_MOUNT_PREFIX=/hostfs -e HOST_PROC=/hostfs/proc telegraf
And this is my test.conf file
[[outputs.file]]
files = ["stdout"]
data_format = "influx"
[[inputs.disk]]
When I run this I get, among many lines about other mount points, the following output.
disk,device=mapper/ubuntu--vg-ubuntu--lv,fstype=ext4,host=140ca42df8ff,mode=rw,path=/ inodes_free=0i,inodes_used=10880i,total=58327040i,free=0i,used=58327040i,used_percent=100,inodes_total=10880i 168 0223760000000000
Which is reporting 100% disk usage on my root mount point disk. Running the disk free utility outside of Docker on the same machine, however, gives a very different and as you might get accurate result.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 1.8T 318G 1.5T 19% /
Is this an issue with Docker volume mounts? This comes directly from the instructions on the linked page on how to use inputs.disk. Why does it think the disk is full?
When I log in to the container with a local shell, I'm able to get the df data myself and I get this very interesting output. It's slightly cropped.
Filesystem Size Used Avail Use% Mounted on
overlay 1.8T 319G 1.5T 19% /
tmpfs 64M 0 64M 0% /dev
tmpfs 24G 0 24G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/loop9 56M 56M 0 100% /hostfs
udev 24G 0 24G 0% /hostfs/dev
tmpfs 24G 0 24G 0% /hostfs/dev/shm
tmpfs 24G 0 24G 0% /hostfs/etc
/dev/mapper/ubuntu--vg-ubuntu--lv 1.8T 319G 1.5T 19% /hostfs/tmp
tmpfs 24G 0 24G 0% /hostfs/sys/fs/cgroup
overlay 1.8T 319G 1.5T 19% /hostfs/var/snap/docker/common/var-lib-docker/overlay2/0d10d0f6cd0032a940715cfec0b9285606c5553bc0ec31f85adf68b30f1b8c83/merged
overlay 1.8T 319G 1.5T 19% /hostfs/var/snap/docker/common/var-lib-docker/overlay2/f7d9f4b0e2fc9d7d6c85513f40802e546ffdcb3f9c000e476d57ecaec0f2f4f0/merged
overlay 1.8T 319G 1.5T 19% /hostfs/var/snap/docker/common/var-lib-docker/overlay2/e8f40d5f2638d2f5f07d97ced86f76934b3780b1e2228a733d08518937726fd1/merged
overlay 1.8T 319G 1.5T 19% /hostfs/var/snap/docker/common/var-lib-docker/overlay2/8737e391698e3aa86a5bd22fe6038753c5ba6330dc83098acd817d0e3e7e45a7/merged
overlay 1.8T 319G 1.5T 19% /hostfs/var/snap/docker/common/var-lib-docker/overlay2/5c3d29d8010c99b64e6235107bd932462fcc07945704f2f286057a3065a18998/merged
tmpfs 4.8G 3.2M 4.8G 1% /hostfs/run
tmpfs 5.0M 0 5.0M 0% /hostfs/run/lock
tmpfs 4.8G 8.0K 4.8G 1% /hostfs/run/user/1001
/dev/sda2 974M 307M 600M 34% /hostfs/var/lib/snapd/hostfs/boot
/dev/loop0 15M 15M 0 100% /hostfs/snap/aws-cli/130
/dev/loop7 1.2M 1.2M 0 100% /hostfs/snap/mosquitto/704
/dev/loop11 1.2M 1.2M 0 100% /hostfs/snap/mosquitto/776
