mdadm : assemble raid5 created with missing disk

Question

2 years ago i've created a raid5 array with 2 disks like this :

mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 missing

Now i've rebuilt my system with latest ubuntu server LTS, and mdadm does not reassemble a running array :

root@htpc:~# mdadm -E /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 530afddc:f4e62791:eba1539c:15672d1d
           Name : ubuntu:0
  Creation Time : Sat Aug 17 11:37:44 2019
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 11720778895 (5588.90 GiB 6001.04 GB)
     Array Size : 11720778752 (11177.81 GiB 12002.08 GB)
  Used Dev Size : 11720778752 (5588.90 GiB 6001.04 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=143 sectors
          State : active
    Device UUID : 1c03a042:a40c1eb8:8aa0533c:22a74311

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Apr  9 17:24:44 2021
  Bad Block Log : 512 entries available at offset 32 sectors
       Checksum : eaaa2b7f - correct
         Events : 223469

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA. ('A' == active, '.' == missing, 'R' == replacing)
root@htpc:~# mdadm -E /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x9
     Array UUID : 530afddc:f4e62791:eba1539c:15672d1d
           Name : ubuntu:0
  Creation Time : Sat Aug 17 11:37:44 2019
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 11720778895 (5588.90 GiB 6001.04 GB)
     Array Size : 11720778752 (11177.81 GiB 12002.08 GB)
  Used Dev Size : 11720778752 (5588.90 GiB 6001.04 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=143 sectors
          State : active
    Device UUID : 019a2ef9:f11076b3:7060bf02:3b6ba98e

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Apr  9 17:24:44 2021
  Bad Block Log : 512 entries available at offset 32 sectors - bad blocks present.
       Checksum : e0624aba - correct
         Events : 223469

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA. ('A' == active, '.' == missing, 'R' == replacing)
root@htpc:~# mdadm --assemble --scan
mdadm: /dev/md/ubuntu:0 assembled from 2 drives - not enough to start the array while not clean - consider --force.
mdadm: No arrays found in config file or automatically
root@htpc:~# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 2
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 2

              Name : ubuntu:0
              UUID : 530afddc:f4e62791:eba1539c:15672d1d
            Events : 223469

    Number   Major   Minor   RaidDevice

       -       8       33        -        /dev/sdc1
       -       8       17        -        /dev/sdb1
root@htpc:~#

There was no sync at all when i created my raid5 array with a missing disk, so it's a "fake" raid5, probably kind of raid0 and i'm not surprised by mdadm assemble output. But i don't want to do any mistake, so what are my options at this point ?

start the array in this state whith

mdam --start /dev/md127

I'm not sure about this as metadata on each disk talks about raid 5 ...

i now have a third disk with a clean /dev/sdd1 partition, should i try

mdadm --assemble /dev/md0 /dev/db1 /dev/sdc1 /dev/sdd1

in this case i think mdadm will try to rebuild the array, but for this mdadm will expect raid 5 striped data on the 2 first disks, but they're not, as it's a kind of raid 0 ...

I really need to recover these datas, but i'm lost in finding the good command. Any help would be appreciated :)

Something like `mdadm --assemble --scan --run --force`. I don't remember which mode you'd use, booting a system with missing drive is very infrequent event, but there aren't too many. Explanation: mdadm won't assemble array automatically if it lost some redundancy since last operation, even if there are still enough devices to run in degraded mode; you have to force it to do so. The last run state is recorded into MD superblock. Probably your case is something like this. If you are concerned (you should be), do `lsblk`, `blkid` and all other relevant checking. (I assumed you already did that.) — Nikita Kipriyanov, Jun 02 '21 at 05:41
`mdadm --assemble --scan --force` did the trick ! Thank you. — Alexandre Barth, Jun 02 '21 at 20:23
I presented that in the form of long and deeply explained answer. Comments might go away, but the answer will stay forever. Please accept it, if you feel it's correct! — Nikita Kipriyanov, Jun 03 '21 at 07:33

score 0 · Answer 1 · answered Jun 03 '21 at 07:32

TL;DR

First is is better to double-check by hand if things are really in expected state and you won't lose any information. This is done by using mdadm --examine --scan and mdadm --examine /dev/sdXY (component), lsblk, blkid, checking dmesg, and so on. From examine commands you'll know UUIDs of arrays you have:

# mdadm --examine --scan
ARRAY /dev/md/0  metadata=1.2 UUID=6bbeca3f:5284fd0a:e15d62ec:b6a5a9a4 name=vh1:0
ARRAY /dev/md/1  metadata=1.2 UUID=cc82d789:94c6c7c2:11e57657:710a4175 name=vh1:1
ARRAY /dev/md/2  metadata=1.2 UUID=ddfbcbfd:9c11fd11:72eab0f2:88e595b1 name=vh1:2

When you are confident you know what you're doing, use --assemble --force to force-assemble a specific array:

mdadm --assemble --force --uuid ddfbcbfd:9c11fd11:72eab0f2:88e595b1 /dev/md/2

or all arrays at once:

mdadm --assemble --force --scan

Have fun!

Explanation

The Linux software RAID tracks which state the array was when it was last seen in operation. It records if the array was working in optimal or degraded state and which devices were present (which array slots were occupied). This information is stored inside MD superblock. The details are described in the MD RAID wiki.

The command mdadm --examine /dev/sdXY will show you the decoded contents of the superblock on the /dev/sdXY, if it is present there:

/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : cc82d789:94c6c7c2:11e57657:710a4175
           Name : vh1:1  (local to host vh1)
  Creation Time : Fri Aug 28 16:54:57 2020
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1953257472 (931.39 GiB 1000.07 GB)
     Array Size : 976628736 (931.39 GiB 1000.07 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=0 sectors
          State : active
    Device UUID : 93be2645:7b75bdb5:89f489ac:003794ff

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Jun  3 10:26:30 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 4c9ddac1 - correct
         Events : 16674


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

This might help to answer some "why" questions.

When MD starts and scans all suitable devices for superblocks, it checks if things really match information that was stored. If something doesn't match, it won't assemble that array automatically. This is done for several reasons, for example, some devices might be slow to initialize, and it is better to wait until everything comes up and start array directly in the optimal state than to start array degraded and then perform recovery once the late device appears.

But if the late device never appears, like that it was physically removed, the array won't start by itself. In this case, you can command it to start by using --force if there is still enough devices for it to run in degraded state.

mdadm : assemble raid5 created with missing disk

1 Answers1

TL;DR

Explanation