Recently replaced a faulty disk in my Raid 6 group. All went as expected, no problem. After getting everything rebuilt using a new /dev/sdi1 as a replacement for the failing /dev/sde1 I decided to try an unplug /dev/sde1. I guessed wrong. I powered down and plugged what I'm now guessing was /dev/sdh's SATA cable back in and powered back up.
/dev/sdh1 is now refusing to be added back into the array, giving me the classic "Device or Resource busy" message when I either try to assemble or hotadd it in.
Digging around, it appears to be confusing /dev/sdh1 with /dev/sdg1:
fs1 etc # mdadm --examine /dev/sdh1
/dev/sdh1:
Magic : a92b4efc
Version : 0.90.00
UUID : 95b6add0:acea3231:a1c8af6a:322d8026
Creation Time : Sat Aug 16 06:40:57 2008
Raid Level : raid6
Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
Array Size : 3662859520 (3493.18 GiB 3750.77 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Sat Dec 7 11:00:34 2013
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : 86c78a1c - correct
Events : 522025
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 97 3 active sync /dev/sdg1
0 0 8 9 0 active sync /dev/sda9
1 1 8 65 1 active sync /dev/sde1
2 2 8 81 2 active sync /dev/sdf1
3 3 8 97 3 active sync /dev/sdg1
4 4 8 33 4 active sync /dev/sdc1
5 5 8 49 5 active sync /dev/sdd1
6 6 8 113 6 active sync /dev/sdh1
Note that "this" is showing as having the same Minor number as /dev/sdg1. Doing an "mdadm --examine" on any other element works as expected, with "this" displaying the minor number that matches the partition.
ugh... looking at all the "mdadm --examines" things look even funkier. Most are showing /dev/sda3 (by minor number) as a fault drive even though /dev/sde1 (minor 65) was the one faulted and removed. /dev/sde1 is now showing as a spare. Up through /dev/sdg1 otherwise look OK, but /dev/sdh1 is reporting as having the same minor number as /dev/sdg1, and /dev/sdi1 is reporting as having the same minor number as /dev/sdh1.
How can I get this corrected? I tried zeroing the front of the /dev/sdh disk and recreating partitions but that didn't help.
I do have a backup of the data, although its on a single disk at the moment. Still, I even attempted to recreate the array, but still get the "device or resource busy":
mdadm --create /dev/md0 --level=6 --raid-devices=7 /dev/sda9 /dev/sdc1 /dev/sdd1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1
mdadm: /dev/sda9 appears to be part of a raid array:
level=raid6 devices=7 ctime=Sat Aug 16 06:40:57 2008
mdadm: /dev/sdc1 appears to be part of a raid array:
level=raid6 devices=7 ctime=Sat Aug 16 06:40:57 2008
mdadm: /dev/sdd1 appears to be part of a raid array:
level=raid6 devices=7 ctime=Sat Aug 16 06:40:57 2008
mdadm: /dev/sdf1 appears to be part of a raid array:
level=raid6 devices=7 ctime=Sat Aug 16 06:40:57 2008
mdadm: /dev/sdg1 appears to be part of a raid array:
level=raid6 devices=7 ctime=Sat Aug 16 06:40:57 2008
mdadm: super1.x cannot open /dev/sdh1: Device or resource busy
mdadm: /dev/sdh1 is not suitable for this array.
mdadm: /dev/sdi1 appears to be part of a raid array :
level=raid6 devices=7 ctime=Sat Aug 16 06:40:57 2008
mdadm: create aborted
