2

We have a CEPH setup with 3 servers and 15 OSDs. Two weeks ago We got "2 OSDs nearly full" warning. We have reweighted the OSD by using below command and restarted both OSDs

ceph osd reweight-by-utilization

After restarting we are getting below warning for the last two weeks

# ceph health detail
HEALTH_WARN Degraded data redundancy: 7 pgs undersized
PG_DEGRADED Degraded data redundancy: 7 pgs undersized
    pg 39.7 is stuck undersized for 1398599.590587, current state active+undersized+remapped, last acting [10,1]
    pg 39.1e is stuck undersized for 1398600.838131, current state active+undersized, last acting [1,10]
    pg 39.2d is stuck undersized for 1398600.848232, current state active+undersized, last acting [10,1]
    pg 39.58 is stuck undersized for 1398600.850871, current state active+undersized, last acting [10,1]
    pg 39.5f is stuck undersized for 1398600.836724, current state active+undersized, last acting [1,10]
    pg 39.79 is stuck undersized for 1398600.848756, current state active+undersized, last acting [10,1]
    pg 54.d is stuck undersized for 1398599.590531, current state active+undersized+remapped, last acting [10,1]



 # ceph -w
  cluster:
    id:     2e7201e4-9cdc-41db-a995-4844eb07c255
    health: HEALTH_WARN
            Degraded data redundancy: 7 pgs undersized

  services:
    mon: 3 daemons, quorum CEPH001,CEPH002,CEPH003
    mgr: CEPH001(active), standbys: CEPH003
    osd: 15 osds: 15 up, 15 in; 10 remapped pgs

  data:
    pools:   11 pools, 1238 pgs
    objects: 292.7 k objects, 1.3 TiB
    usage:   4.0 TiB used, 41 TiB / 45 TiB avail
    pgs:     1223 active+clean
             8    active+clean+remapped
             5    active+undersized
             2    active+undersized+remapped

  io:
    client:   21 KiB/s rd, 1.1 MiB/s wr, 55 op/s rd, 100 op/s wr

I am new to CEPH, is this expected behavior? Or how to resolve this?

    # ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 1   ssd 1.00000  0.95001 447 GiB 5.8 GiB 441 GiB  1.29 0.14 166
 5   ssd 1.00000  0.95001 447 GiB 5.8 GiB 441 GiB  1.29 0.14 159
10   ssd 1.00000  0.95001 447 GiB  71 GiB 376 GiB 15.96 1.78 166
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 1   ssd 1.00000  0.95001 447 GiB 5.8 GiB 441 GiB  1.29 0.14 166
 5   ssd 1.00000  0.95001 447 GiB 5.8 GiB 441 GiB  1.29 0.14 159
10   ssd 1.00000  0.95001 447 GiB  71 GiB 376 GiB 15.96 1.78 166
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 0   hdd 1.00000  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 1.00000  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 1.00000  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 1.00000  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 6   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 1.00000  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 1.00000  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 1.00000  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
11   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 1.00000  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 1.00000  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 1.00000  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
 0   hdd 3.63899  1.00000 3.6 TiB 347 GiB 3.3 TiB  9.31 1.04 287
 2   hdd 3.63899  1.00000 3.6 TiB 350 GiB 3.3 TiB  9.39 1.05 266
 3   hdd 3.63899  1.00000 3.6 TiB 307 GiB 3.3 TiB  8.25 0.92 255
 4   hdd 3.63899  1.00000 3.6 TiB 363 GiB 3.3 TiB  9.75 1.09 286
 1   ssd 1.00000  0.95001 447 GiB 5.8 GiB 441 GiB  1.29 0.14 166
 6   hdd 3.63899  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.61 1.07 274
 7   hdd 3.63899  1.00000 3.6 TiB 369 GiB 3.3 TiB  9.91 1.11 270
 8   hdd 3.63899  1.00000 3.6 TiB 317 GiB 3.3 TiB  8.51 0.95 242
 9   hdd 3.63899  1.00000 3.6 TiB 358 GiB 3.3 TiB  9.62 1.07 254
 5   ssd 1.00000  0.95001 447 GiB 5.8 GiB 441 GiB  1.29 0.14 159
11   hdd 3.63899  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.88 0.99 277
12   hdd 3.63899  1.00000 3.6 TiB 331 GiB 3.3 TiB  8.89 0.99 269
13   hdd 3.63899  1.00000 3.6 TiB 279 GiB 3.4 TiB  7.49 0.84 260
14   hdd 3.63899  1.00000 3.6 TiB 330 GiB 3.3 TiB  8.85 0.99 276
10   ssd 1.00000  0.95001 447 GiB  71 GiB 376 GiB 15.96 1.78 166
                    TOTAL  45 TiB 4.0 TiB  41 TiB  8.96
MIN/MAX VAR: 0.14/1.78  STDDEV: 2.03

# ceph osd tree
ID  CLASS WEIGHT   TYPE NAME                STATUS REWEIGHT PRI-AFF
-43       12.00000 pool demognocchi
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-40       12.00000 pool demobackup
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-37        3.00000 pool demossd
  1   ssd  1.00000     osd.1                    up  0.95001 1.00000
  5   ssd  1.00000     osd.5                    up  0.95001 1.00000
 10   ssd  1.00000     osd.10                   up  0.95001 1.00000
-34       12.00000 pool demosata
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-31       12.00000 pool demoglance
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-18       12.00000 pool defaultbackup
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-17       12.00000 pool backup
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-16       12.00000 pool gnocchi
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-15        3.00000 pool ssdvolume01
  1   ssd  1.00000     osd.1                    up  0.95001 1.00000
  5   ssd  1.00000     osd.5                    up  0.95001 1.00000
 10   ssd  1.00000     osd.10                   up  0.95001 1.00000
-14       12.00000 pool defaultsata01
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
-13       12.00000 pool defaultglance01
  0   hdd  1.00000     osd.0                    up  1.00000 1.00000
  2   hdd  1.00000     osd.2                    up  1.00000 1.00000
  3   hdd  1.00000     osd.3                    up  1.00000 1.00000
  4   hdd  1.00000     osd.4                    up  1.00000 1.00000
  6   hdd  1.00000     osd.6                    up  1.00000 1.00000
  7   hdd  1.00000     osd.7                    up  1.00000 1.00000
  8   hdd  1.00000     osd.8                    up  1.00000 1.00000
  9   hdd  1.00000     osd.9                    up  1.00000 1.00000
 11   hdd  1.00000     osd.11                   up  1.00000 1.00000
 12   hdd  1.00000     osd.12                   up  1.00000 1.00000
 13   hdd  1.00000     osd.13                   up  1.00000 1.00000
 14   hdd  1.00000     osd.14                   up  1.00000 1.00000
 -1       46.66498 root default
 -3       15.55499     host CEPH001
  0   hdd  3.63899         osd.0                up  1.00000 1.00000
  2   hdd  3.63899         osd.2                up  1.00000 1.00000
  3   hdd  3.63899         osd.3                up  1.00000 1.00000
  4   hdd  3.63899         osd.4                up  1.00000 1.00000
  1   ssd  1.00000         osd.1                up  0.95001 1.00000
 -7       15.55499     host CEPH002
  6   hdd  3.63899         osd.6                up  1.00000 1.00000
  7   hdd  3.63899         osd.7                up  1.00000 1.00000
  8   hdd  3.63899         osd.8                up  1.00000 1.00000
  9   hdd  3.63899         osd.9                up  1.00000 1.00000
  5   ssd  1.00000         osd.5                up  0.95001 1.00000
-10       15.55499     host CEPH003
 11   hdd  3.63899         osd.11               up  1.00000 1.00000
 12   hdd  3.63899         osd.12               up  1.00000 1.00000
 13   hdd  3.63899         osd.13               up  1.00000 1.00000
 14   hdd  3.63899         osd.14               up  1.00000 1.00000
 10   ssd  1.00000         osd.10               up  0.95001 1.00000

# ceph osd pool ls detail
pool 37 'defaultglance01' replicated size 3 min_size 1 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 128 last_change 2993 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
        removed_snaps [1~3,7~2e,37~4,41~a,50~4,56~2,5a~2,5d~3,61~1]
pool 38 'defaultsata01' replicated size 3 min_size 1 crush_rule 3 object_hash rjenkins pg_num 200 pgp_num 200 last_change 2971 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
        removed_snaps [1~59]
pool 39 'ssdvolume01' replicated size 3 min_size 1 crush_rule 1 object_hash rjenkins pg_num 150 pgp_num 150 last_change 3005 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
        removed_snaps [1~3]
pool 40 'gnocchi' replicated size 3 min_size 1 crush_rule 5 object_hash rjenkins pg_num 256 pgp_num 256 last_change 1821 flags hashpspool stripe_width 0 application rbd
pool 45 'backup' erasure size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 256 pgp_num 256 last_change 2392 flags hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 8192 application rbd
        removed_snaps [1~b]
pool 46 'defaultbackup' replicated size 3 min_size 1 crush_rule 7 object_hash rjenkins pg_num 56 pgp_num 56 last_change 2098 flags hashpspool stripe_width 0 application rbd
pool 52 'demoglance' replicated size 3 min_size 1 crush_rule 8 object_hash rjenkins pg_num 16 pgp_num 16 last_change 2848 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
        removed_snaps [1~3,7~1,9~1,b~2]
pool 53 'demosata' replicated size 3 min_size 1 crush_rule 9 object_hash rjenkins pg_num 128 pgp_num 128 last_change 2974 lfor 0/2620 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
        removed_snaps [1~6,8~f,1d~2,20~2]
pool 54 'demossd' replicated size 3 min_size 1 crush_rule 10 object_hash rjenkins pg_num 16 pgp_num 16 last_change 3005 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
        removed_snaps [1~3]
pool 55 'demobackup' replicated size 3 min_size 1 crush_rule 11 object_hash rjenkins pg_num 16 pgp_num 16 last_change 2596 flags hashpspool stripe_width 0 application rbd
pool 56 'demognocchi' replicated size 3 min_size 1 crush_rule 12 object_hash rjenkins pg_num 16 pgp_num 16 last_change 2597 flags hashpspool stripe_width 0 application rbd

ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)

adminz
  • 215
  • 1
  • 2
  • 13
  • Can you edit your question and add `ceph osd df` output? I assume that the reweight doesn't have enough space available and can't finish successfully. Your only options are either to expand the cluster and add more storage or delete unused data to clear those warnings. – eblock Jan 07 '21 at 08:47
  • @eblock updated. reweight executed successfully and have enough space available – adminz Jan 07 '21 at 10:00
  • Please read my comment again, the command I requested was `ceph osd df`. Are you using pool size 2? That's not recommended if you value your data, for testing purposes it's fine. Do you have any custom rules in place? If so also paste them in your question. – eblock Jan 07 '21 at 10:32
  • @eblock Sorry updated results. We are using 12 disks (4 disk in each server). – adminz Jan 07 '21 at 11:54
  • 1
    Please also add `ceph osd tree` to the question. Can you explain how the SSDs are used? – eblock Jan 07 '21 at 12:02
  • @eblock updated with requested results. SSDs are a separate pool with 3 SSD disks. For we can provide premium volume. – adminz Jan 07 '21 at 12:16
  • 1
    I'm not even sure how you did that with the osd tree having a per pool output, too? Which ceph version are you running on? Which pools are 39 and 54 (`ceph osd pool ls detail`)? – eblock Jan 07 '21 at 12:24
  • @eblock Results added. pools 39 and 54 are for SSD, we used for two different Infra. ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable) – adminz Jan 07 '21 at 12:32
  • 3
    I still don't get the whole picture. which ceph version is this? I'm wondering why the crush weight differs between per pool output and in the regular osd tree output. Anyway, I would try to reweight the SSDs back to 1, there's no point in that if you have 3 SSDs but reduce all of the reweights equally. What happens if you run `ceph osd crush reweight osd.1 1` and repeat that for the other two SSDs? – eblock Jan 07 '21 at 12:42
  • @eblock ssdvolume01 & demossd two pools for using SSD volume for two infra. we have three SSD disks spread across three servers. The SSD were weighted 1 it changed after executing #ceph osd reweight-by-utilization. How can I change back to 1 without losing data. – adminz Jan 07 '21 at 12:52
  • 1
    A reweight does not purge any data, it just reshuffles it. So if you execute the command I suggested nothing will happen to your data. – eblock Jan 07 '21 at 12:58
  • @eblock I will try and update the result – adminz Jan 07 '21 at 13:02
  • @eblockThe weight mentioned in our crush map is 1 so we used $ceph osd reweight 1 and resolved. Thank you for the help – adminz Jan 09 '21 at 06:16
  • Hate to mention it guys, but this is NOT a programming question. And it is mistagged because it is not related to OpenStack or Ubuntu. – Stephen C Mar 18 '23 at 07:09

1 Answers1

0

We are using crush map and the OSD pool weight mentioned in our Crush map is 1 So we reweighted SSD OSDs with one.

#ceph osd reweight <OSD Number> 1

Thank you @eblock for helping.

adminz
  • 215
  • 1
  • 2
  • 13