Drive replaced in RAID5 not automatically syncing, instead showing as spare

Question

Replacing the drives in a RAID5. sdb failed, replaced it. Things went well. Now I'm replacing sda, but after being added to the array it shows as a spare and doesn't sync.

Steps followed:

mdadm --manage /dev/md127 --fail /dev/sda1.
mdadm --manage /dev/md127 --remove /dev/sda1.
Turn off system. Replace sda drive.
Partition as gpt using parted with raid flag.
mdadm --manage /dev/md127 --add /dev/sda1.
cat proc/mdstat to check that we're syncing. We're not.

I can't figure out why we're not syncing. Any help would be appreciated. Output is shown below.

RAID details:

[me@me /]# mdadm --detail /dev/md127
/dev/md127:
        Version : 1.1
  Creation Time : Mon Oct 22 16:20:37 2012
     Raid Level : raid5
     Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 976759296 (931.51 GiB 1000.20 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon Nov  7 18:33:10 2016
          State : clean, degraded
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

  Delta Devices : 1, (3->4)

           Name : meme:0
           UUID : 28cf18b6:b05b9701:5d28754b:c387cb95
         Events : 247368

    Number   Major   Minor   RaidDevice State
       3       8       49        0      active sync   /dev/sdd1
       2       0        0        2      removed
       4       8       33        2      active sync   /dev/sdc1
       6       8       17        3      active sync   /dev/sdb1

       5       8        1        -      spare   /dev/sda1

MD stat:

[me@me /]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sda1[5](S) sdc1[4] sdd1[3] sdb1[6]
      1953518592 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/3] [U_UU]
      bitmap: 4/8 pages [16KB], 65536KB chunk

unused devices: <none>

Update

There is data on the RAID. It's currently mounted and being used. I don't mind disabling it while I'm replacing the drives, but I'd like to preserve the data on the array.

Update

Worked around the headache. rsynced the contents of the degraded array to a backup drive, replaced the RAID with fresh-baked drives, then rsynced back. In the process, we learned the value of scheduling automated incremental backups (as opposed to the schedule being based on "Hey when was the last time we backed up the system? Oh God do it now!")

duplicate of http://serverfault.com/questions/676638/mdadm-drive-replacement-shows-up-as-spare-and-refuses-to-sync — Ipor Sircer, Nov 07 '16 at 23:52
@IporSircer Tried that solution. Did not work. My `sync_action` file reads '"idle" — Voriki, Nov 07 '16 at 23:53
in worst case you can recreate the array: http://www.kossboss.com/sparetoactive (be careful!) — Ipor Sircer, Nov 07 '16 at 23:54
I never use `--manage`, and it's not mentioned in my manpage. Try removing the device: `mdadm /dev/md127 -f /dev/sda1 -r /dev/sda1`, clearing the metadata: `mdadm --zero-superblock /dev/sda1` and adding it again: `mdadm /dev/md127 -a /dev/sda1` — wurtel, Nov 08 '16 at 14:02
yes, as @wurtel said you have to clean the superblock. After that re-adding should do the trick. — Str82DHeaD, Nov 08 '16 at 14:42
@wurtel I'm a bit new to this, but wouldn't clearing the metadata lead to the data on the array being unrecoverable? In hindsight, I realize I should've mentioned the drives aren't empty. Added it to the question now. — Voriki, Nov 08 '16 at 20:14
@Mirov sda disk haven't any data. Don't afraid clear it before it will be joined to the array. If that disk was used in other RAID it has RAID metadata witch doesn't allow use it in new array. You need to clean only sda. — Mikhail Khirgiy, Nov 09 '16 at 05:23
What Mikhail said... you stated that sda was replaced, i.e. contains NO data. I never mentioned any drive besides sda! — wurtel, Nov 09 '16 at 07:42
@wurtel That's a good point. Overlooked that. Thanks guys. I'll try that tonight. — Voriki, Nov 09 '16 at 14:23

Drive replaced in RAID5 not automatically syncing, instead showing as spare

0 Answers0