0

In one of our servers one of the disk failed, no worries it happens all the time, we replaced the bad disk following the usual procedure but then ... 'the horror', after adding a new disk one of the partition failed to sync because the 'good' disk now active contains some bad blocks, in the details it says 'rebuilding':

Number   Major   Minor   RaidDevice State
   0       8        3        0      active sync   /dev/sda3
   2       8       19        1      spare rebuilding   /dev/sdb3

But the cat to /proc/mdstat show that it already finished, but wasn't added.

md2 : active raid1 sdb3[2] sda3[0]
  1458830400 blocks [2/1] [U_]

I found this:

RAID 1 disk is marked as Spare after replacement

Is this the only/recommended way of proceeding now?, It is not possible to force the rebuild ignoring bad block to then replace the old disk?. Should I try with e2fsck ?

P.S. Off course everything is backed up, but set up all the services in the server might be painful.

Thank you very much in advance.

Community
  • 1
Agustin Cautin
  • 113
  • 4

1 Answers1

1

If your "good" drives have unrecoverable read errors (URE's), and there is no redundancy in your remaining drives, then there is nowhere to find the missing data to rebuild the RAID.

Sorry but this is time to (being cautious, run diags on old drive if you have time) set up new RAID, then install OS then restore data. For future you may want to consider also backing up your system state so this can also be restored.

Once unrecommended way I can think of to fix this - boot from a live CD such as Acronis True Image Rescue CD, then use this to take an image of the existing drive, telling it to ignore bad sectors. Then build your RAID and restore the created image. I don't like this idea as Acronis has no way to regenerate the missing data and you have no idea if/when that will bite you.

I'm guessing the question hasn't been flooded with answers except for womble as most users here have little sympathy for those who wish to restore data they hadn't backed up (i.e. your system state) - RAID does not equal backup.

Robin Gill
  • 2,513
  • 14
  • 13
  • Please forgive my ignorance, but can you explain me more about the system state backup. Because for me the system was backed up :p. Thanks. – Agustin Cautin Jul 13 '12 at 11:33
  • When I've had to back up a server, I've had to make the call whether we need to back up just the data or whole system state + data. Just data is obviously just your important data. The idea being that in the event of a restore, you set up a new OS and other server apps, then just import your data. For a full system state backup, you need to backup the operating system and all programs/files. The problem is to create a system state you cannot just copy a folder as you need software that is capable of copying various files that will be locked by windows. – Robin Gill Jul 13 '12 at 12:04
  • Here's a bit more of an explaination by someone else http://www.datamills.com/Tutorials/systemstate/tutorial.htm When I've needed to do a system state backup I've always used Backup Exec for online systems, or if the system is offline then I've used Acronis True Image rescue CD's. It appears it is possible using ntbackup, but I've never tried this. http://technet.microsoft.com/library/cc781353.aspx – Robin Gill Jul 13 '12 at 12:06
  • Also to note, for a system state backup, you may not need to actually back up the entire drive(s), for example you may have folders that you know are just useless files that will not need to be backed up, and you may wish to seperate your data backup from the system state/entire backup. – Robin Gill Jul 13 '12 at 12:08
  • Final note - thanks but, don't get so eager to accept an answer as best on here. People may answer your question hours or maybe even a day later, other people may have better input but may not bother as the question's got an accepted answer. – Robin Gill Jul 13 '12 at 12:10
  • Wow, thank you so much for taking the time with this valuable advices, I should have mentioned that this are ubuntu servers, but of course the concepts are the same. – Agustin Cautin Jul 13 '12 at 12:31