1

I'm trying to understand how to find out the current and real disk usage of a ceph cluster and I noticed that the output of a rbd du is way different from the output of a df -h once that rbd is mounted as a disk.

Example:

Inside the ToolBox I have the following:

$ rbd du replicapool/csi-vol-da731ad9-eebe-11eb-9fbd-f2c976e9e23a
warning: fast-diff map is not enabled for csi-vol-da731ad9-eebe-11eb-9fbd-f2c976e9e23a. operation may be slow.
2021-09-01T13:53:23.482+0000 7f8c56ffd700 -1 librbd::object_map::DiffRequest: 0x557402c909c0 handle_load_object_map: failed to load object map: rbd_object_map.8cdeb6e704c7e0
NAME                                          PROVISIONED  USED
csi-vol-da731ad9-eebe-11eb-9fbd-f2c976e9e23a      100 GiB  95 GiB

But, inside the Pod that is mounting this rbd, I have:

$ k exec -it -n monitoring prometheus-prometheus-operator-prometheus-1 -- sh
Defaulting container name to prometheus.
Use 'kubectl describe pod/prometheus-prometheus-operator-prometheus-1 -n monitoring' to see all of the containers in this pod.
/prometheus $ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  38.0G     19.8G     18.2G  52% /
...
/dev/rbd5                97.9G     23.7G     74.2G  24% /prometheus
...

Is there a reason for the two results to be so different? Can this be a problem when ceph keeps track of the total space used by the cluster to know how much space is available?

AndD
  • 2,281
  • 1
  • 8
  • 22
  • I would ask this question at the official GitHub repo of [Ceph RBD CSI driver](https://github.com/ceph/ceph-csi). You can create a new issue there and ask about this discrepancy. – mario Sep 14 '21 at 19:08
  • Can it be because ceph triplicates objects in the OSD backend? In my case ceph -s outputs 32 GiB in objects, but the total usage is 89 GiB. I did notice that the usage is around 3x to 4x depending on the object sizes – B Abali Nov 05 '21 at 20:02
  • 1
    I actually found the reason by asking in ceph Slack Channel. I'm not really expert enough to explain in details but more or less, it depends by the fact that the rbd are mounted without the discard flag. This means that if a file is written inside the rbd and later removed, that size will still count as modified by ceph. Which means that if the discarded flag is not used, all the rbd will converge the used size to their maximum size eventually. – AndD Nov 07 '21 at 09:59

0 Answers0