0

When "rados list-inconsistent-pg " returns several inconsistent pgs such as ["1.12","1.2c","1.2d","1.57"], "rados list-inconsistent-obj " gives "Unknown error (1) Operation not permitted".

What should I do to correct his error?

-- update:
"ceph health detail" returns:

HEALTH_ERR 8 scrub errors; Possible data damage: 4 pgs inconsistent
OSD_SCRUB_ERRORS 8 scrub errors
PG_DAMAGED Possible data damage: 4 pgs inconsistent
    pg 1.12 is active+clean+inconsistent, acting [2,1]
    pg 1.2c is active+clean+scrubbing+deep+inconsistent+repair, acting [2,1]
    pg 1.2d is active+clean+inconsistent, acting [2,0]
    pg 1.57 is active+clean+inconsistent, acting [1,2]
James Yu
  • 399
  • 6
  • 22
  • lets try to solve it. First thing first, what does your `sudo ceph health detail` says? – BZKN Feb 10 '22 at 06:36
  • 1
    @BKN-بیکےاین: I append update info to the post – James Yu Feb 10 '22 at 06:57
  • Another thing to add is that if you really have a pool size of 2 you'll probably be facing difficulties repairing those PGs. If there are only two copies, how can ceph determine which one is the correct one? I advise to increase to pool size = 3 in order to avoid that or at least have better chances when trying to repair PGs. Do other ceph/rados commands work as the same user? Or do you run all commands with the admin client & keyring? – eblock Feb 10 '22 at 13:50

1 Answers1

0

Looking at your sudo ceph health detail, first thing is to get help of following general repair command:

ceph pg repair {pgid}

In your case it would be:

  • ceph pg repair 1.12
  • ceph pg repair 1.2c
  • ceph pg repair 1.2d
  • ceph pg repair 1.57

Also, please try to backup the PG copies you are repairing.

Limitations of Ceph repair: My experience says that Ceph repair does not automatically repair placement groups when inconsistencies are found in them. For instance this also explains why repair will not automatically solve it. There are chances you have to manually diagnose like this.

Dig deeper: In addition to it, I maintain following helpful notes for solving PG inconsistencies in my cluster:

BZKN
  • 1,499
  • 2
  • 10
  • 25