health: HEALTH_ERR - how to fix it without losing data?

Question

got ceph status:

# ceph status
  cluster:
    id:     b683c5f1-fd15-4805-83c0-add6fbb7faae
    health: HEALTH_ERR
            1 backfillfull osd(s)
            8 pool(s) backfillfull
            50873/1090116 objects misplaced (4.667%)
            Degraded data redundancy: 34149/1090116 objects degraded (3.133%), 3 pgs degraded, 3 pgs undersized
            Degraded data redundancy (low space): 6 pgs backfill_toofull

  services:
    mon: 3 daemons, quorum tb-ceph-2-prod,tb-ceph-4-prod,tb-ceph-3-prod
    mgr: tb-ceph-1-prod(active)
    osd: 6 osds: 6 up, 6 in; 6 remapped pgs
    rgw: 4 daemons active

  data:
    pools:   8 pools, 232 pgs
    objects: 545.1 k objects, 153 GiB
    usage:   728 GiB used, 507 GiB / 1.2 TiB avail
    pgs:     34149/1090116 objects degraded (3.133%)
             50873/1090116 objects misplaced (4.667%)
             226 active+clean
             3   active+undersized+degraded+remapped+backfill_toofull
             3   active+remapped+backfill_toofull

  io:
    client:   286 KiB/s rd, 2 op/s rd, 0 op/s wr

Here is OSD statuses:

# ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS
 2   hdd 0.09769  1.00000 100 GiB  32 GiB  68 GiB 32.38 0.55  30
 5   hdd 0.32230  1.00000 330 GiB 220 GiB 110 GiB 66.71 1.13 122
 0   hdd 0.32230  1.00000 330 GiB 194 GiB 136 GiB 58.90 1.00 125
 1   hdd 0.04390  0.95001  45 GiB  43 GiB 2.5 GiB 94.53 1.60  11
 3   hdd 0.09769  1.00000 100 GiB  42 GiB  58 GiB 42.37 0.72  44
 4   hdd 0.32230  0.95001 330 GiB 196 GiB 134 GiB 59.43 1.01 129
                    TOTAL 1.2 TiB 728 GiB 507 GiB 58.94
MIN/MAX VAR: 0.55/1.60  STDDEV: 19.50

I have tried these commands:

 ceph osd pool set default.rgw.buckets.data pg_num 32
 ceph osd pool set default.rgw.buckets.data pgp_num 32

But it didn't help either. I think pg_num 32 is too small for my OSD count, but not sure if it's safe to set it bigger while health status is err

score 0 · Answer 1 · answered Mar 02 '20 at 15:55

Your OSD #1 is full. The disk drive is fairly small and you should probably exchange it with a 100G drive like the other two you have in use. To remedy the situation have a look at the Ceph control commands.

The command ceph osd reweight-by-utilization will adjust the weight for overused OSDs and trigger rebalance of PGs. See also this blog post describing this situation.

health: HEALTH_ERR - how to fix it without losing data?

1 Answers1