I am new user of Ceph.
My Ceph server is running version 12.2.5. It has 3 nodes and each node is having 1 OSD with 4TB HDD.
The ceph.conf
osd pool default size = 3
osd pool default min size = 2
I have upload 252GB data to Ceph.
For my understanding, the raw storage usage should be around 3 times because of setting osd pool default size = 3
.
But now my global raw storage usage is almost 6 times of my data (1605GB / 252GB = 6.3)
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
11165G 9560G 1605G 14.38
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
.rgw.root 8 1113 0 3000G 4
default.rgw.control 9 0 0 3000G 8
default.rgw.meta 10 21502 0 3000G 72
default.rgw.log 11 0 0 3000G 269
default.rgw.buckets.index 15 0 0 3000G 427
default.rgw.buckets.data 16 256G 7.88 3000G 6522963
Is anyone have the same problem on Ceph?
Update
I think I find out the reason. Most of my file is a small file, but in Ceph version 12.2.5 (bluestore), bluestore have minimum allocate size bluestore_min_alloc_size_hdd = 65536
, therefore if the file is small than 64k, it still use 64k to store it.
I tested upload 2000 files (each 64k) and 3200 files (each 40k), both total around 128M
The raw usage of 64k file is 410M, close to 384M (2000 * 64k * 3)
The raw usage of 40k file is 602M, close to 614M (3200 * 64k * 3)