Recover mdadm RAID 10 array: listing all as spares

Question

I'm a bad person and haven't backed up my RAID elsewhere. I now have a RAID10 array that won't assemble, and I'm hopeful I can save it. Here's the details:

I have five hard drives set up as RAID10 (4+1 Spare). For unknown reasons, two failed and my spare was broken, and now I can't seem to reassemble them.

I have some information below, notably the output of mdstat after a reboot [1], the output of stopping the array and attempting to reassemble it [2] and detailed information from mdadm -E [3].

The literature seems to indicate that I can likely force recreate the RAID array using --create and --assume-clean, but I'm worried it will aggravate my data.

My current thoughts are to recreate the array in a degraded state using the drives with the highest matching events (sdc1, sdd1 and sde1, below). Is there a better solution?

megatron ~ # mdadm -E /dev/sd[bcdefghijklmnop]1 | egrep 'Event|/dev'
/dev/sdb1:
         Events : 494734
/dev/sdc1:
         Events : 502154
/dev/sdd1:
         Events : 502154
/dev/sde1:
         Events : 502154
/dev/sdf1:
         Events : 494756

[1] This is the output of mdstat, after a reboot:

megatron ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid10] [raid6] [raid5] [raid4]
md0 : inactive sde1[3](S) sdf1[2](S) sdc1[1](S) sdd1[4](S) sdb1[0](S)
      7814052444 blocks super 1.2

unused devices: <none>

[2] If I stop the array and reassemble it, I get the following:

megatron ~ # mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sde2
mdadm: /dev/sde2 has wrong uuid.
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sde has wrong uuid.
mdadm: no RAID superblock on /dev/sdf
mdadm: /dev/sdf has wrong uuid.
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdd has wrong uuid.
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdc has wrong uuid.
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda3: Device or resource busy
mdadm: /dev/sda3 has wrong uuid.
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0.
mdadm: added /dev/sdb1 to /dev/md0 as 0
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: added /dev/sdf1 to /dev/md0 as -1
mdadm: added /dev/sdc1 to /dev/md0 as -1
mdadm: added /dev/sdd1 to /dev/md0 as 2
mdadm: /dev/md0 assembled from 2 drives and 2 spares - not enough to start the array.

[3] Lastly, the output of mdadm -E /dev/sd[bcdefghijklmnop]1 yields:

/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
           Name : megatron:1  (local to host megatron)
  Creation Time : Thu Feb  3 23:38:56 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
     Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
  Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 09ce480d:aad5f0ac:4cfbc777:7303d4a7

    Update Time : Tue Jul  9 20:16:22 2013
       Checksum : 8a17d4c9 - correct
         Events : 494734

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
           Name : megatron:1  (local to host megatron)
  Creation Time : Thu Feb  3 23:38:56 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
     Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
  Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : d89673c0:5efc51a7:1589bbc2:d7002e7d

    Update Time : Wed Jul 10 00:14:04 2013
       Checksum : 57efa473 - correct
         Events : 502154

         Layout : near=2
     Chunk Size : 512K

   Device Role : spare
   Array State : ..AA ('A' == active, '.' == missing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
           Name : megatron:1  (local to host megatron)
  Creation Time : Thu Feb  3 23:38:56 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
  Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 08542b46:f4cfa093:31650e72:dd781e74

    Update Time : Wed Jul 10 00:14:04 2013
       Checksum : aa71a2c3 - correct
         Events : 502154

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : ..AA ('A' == active, '.' == missing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
           Name : megatron:1  (local to host megatron)
  Creation Time : Thu Feb  3 23:38:56 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
     Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
  Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 8c655b9d:4630e93b:32eb61b8:f7a5f513

    Update Time : Wed Jul 10 00:14:04 2013
       Checksum : 55dcae05 - correct
         Events : 502154

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : ..AA ('A' == active, '.' == missing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
           Name : megatron:1  (local to host megatron)
  Creation Time : Thu Feb  3 23:38:56 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
     Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
  Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 3b54d74e:a1065dd6:d3e836ca:b340b56d

    Update Time : Tue Jul  9 20:17:04 2013
       Checksum : d63b6fa - correct
         Events : 494756

         Layout : near=2
     Chunk Size : 512K

   Device Role : spare
   Array State : .AAA ('A' == active, '.' == missing)

I wish I could help you with your question, but as a note: it's more friendly to use long-options when specifying command line params. — Halfgaar, Jul 10 '13 at 11:42
BTW, if you have drives to spare; dd them over and experiment with those drives. Then it doesn't matter if you mess it up. — Halfgaar, Jul 10 '13 at 11:43

score 1 · Answer 1 · answered Aug 06 '13 at 15:51

Wrong two drives failed in RAID10 4+1 configuration will break the array, yes :(

It looks lucky for you that three of them ARE in sync (as you say, with 502154), so yes, you should try to reassemble three of them. It looks pretty hopefull.

Of course, if you had 4 extra drives you should make copies before trying in case it does not go so well...

Recover mdadm RAID 10 array: listing all as spares

1 Answers1