3

I am new user of Ceph.

My Ceph server is running version 12.2.5. It has 3 nodes and each node is having 1 OSD with 4TB HDD.

The ceph.conf

osd pool default size     = 3
osd pool default min size = 2

I have upload 252GB data to Ceph.

For my understanding, the raw storage usage should be around 3 times because of setting osd pool default size = 3.

But now my global raw storage usage is almost 6 times of my data (1605GB / 252GB = 6.3)

GLOBAL:
    SIZE       AVAIL     RAW USED     %RAW USED
    11165G     9560G        1605G         14.38
POOLS:
    NAME                          ID     USED      %USED     MAX AVAIL     OBJECTS
    .rgw.root                     8       1113         0         3000G           4
    default.rgw.control           9          0         0         3000G           8
    default.rgw.meta              10     21502         0         3000G          72
    default.rgw.log               11         0         0         3000G         269
    default.rgw.buckets.index     15         0         0         3000G         427
    default.rgw.buckets.data      16      256G      7.88         3000G     6522963

Is anyone have the same problem on Ceph?

Update

I think I find out the reason. Most of my file is a small file, but in Ceph version 12.2.5 (bluestore), bluestore have minimum allocate size bluestore_min_alloc_size_hdd = 65536, therefore if the file is small than 64k, it still use 64k to store it.

I tested upload 2000 files (each 64k) and 3200 files (each 40k), both total around 128M

The raw usage of 64k file is 410M, close to 384M (2000 * 64k * 3)

The raw usage of 40k file is 602M, close to 614M (3200 * 64k * 3)

Kaden Sy
  • 31
  • 4

1 Answers1

0

I am having the same issue. I have only realised this after creating the pool. You can change the min_alloc to 4096 but I don't know how that will perform for you. I would advise though to go halfway to 32k or 16k but not lower.

Vish
  • 4,508
  • 10
  • 42
  • 74
  • I have the exact same issue, changing `bluestore_min_alloc_size` doesn't seem to do anything, i.e. config shows the new value, but files effectively still use 64K. What am I missing? – Tony Bogdanov Sep 05 '20 at 21:57
  • I think that the only solution would be to create a new cluster and migrate to that. Also, I wouldn't recommend storing anything below 16k as you might as well use a SSD for that size objects. – Vish Oct 18 '20 at 23:23
  • The underlying block storage **is** an SSD and 50% of my files are below 1KB anyway.. – Tony Bogdanov Oct 19 '20 at 05:23
  • Thats a lot of small objects. If you have more than 10% of the files as less than 64kb, just use a ssd for direct storage (much faster with nonexistent overhead). Use standard raid or just use zfs mirror. Ceph is not suited for small objects size I have found. Its more suitable for larger objects. By default the block size is 4k on ssd's, so an upgrade to octopus might work for you. – Vish Oct 19 '20 at 09:00
  • I know Ceph is not meant for small files, but it is by far the most robust and (actually) fastest solution (memory caching?) for a distributed network storage in a Kubernetes cluster. I can't use SSD, I need the storage to be shared (and simultaneously accessible) accross multiple machines in different physical locations. – Tony Bogdanov Oct 20 '20 at 12:34