3

Over the weekend two drives failed on one of our development machines and I lost a lot of work. The machine has six 300gb drives in a raid 5 array. Our system administrator tells me that if only one drive failed it would be possible to recover. We've tried putting the drives in the freezer for a few hours with no luck. It sounds like sending the array off to a recovery specialist would be very costly and hit or miss. Any ideas?

Zypher
  • 37,405
  • 5
  • 53
  • 95
Daniel
  • 169
  • 1
  • 8
  • 1
    Freezer? really? – Chopper3 Jan 19 '10 at 20:30
  • You've got me. SA said he's tried it in emergencies and it's worked occasionally. – Daniel Jan 19 '10 at 20:35
  • 1
    The freezer trick has worked for me. At least long enough to get what I needed to get off it. Granted it only a few times it has worked. – aduljr Jan 19 '10 at 20:49
  • 2
    I've sucessfully used the freezer trick too. It's a last-ditch-nothing-else-has-worked trick. And it only works once. – Tom O'Connor Jan 19 '10 at 20:53
  • 1
    The freezer bit has to do with the thingies and diddly-bobs (probably bearings?) failing as they thermally expand. The freezing doesn't, so much "magically" fix the problem as it provides a *longer* expansion period before failure. It is, without question, a last ditch effort, but I have used it successfully during data recovery jobs. The trick is to get it nice and cold, then dupe it as fast as possible. – Scott Pack Jan 17 '12 at 03:41

5 Answers5

11

If you've lost two drives then you've lost your data. The only recourse is to restore from backup if one exists or, as you stated, send the drives off to a data recovery company to try and recover the data.

joeqwerty
  • 109,901
  • 6
  • 81
  • 172
  • What an unfortunate twist of fate to lose two drives in one weekend. The server is designed to be an exact copy of the on-site server, though unfortunately this one is sans backup. – Daniel Jan 19 '10 at 19:45
  • 1
    In that case then why not replace the failed drives and sync the content from the on-site server? – joeqwerty Jan 19 '10 at 19:50
  • 2
    And when you replace the drives give some serious consideration to adding a hot spare, or creating a RAID 6 array instead. – John Gardeniers Jan 19 '10 at 20:23
  • 4
    @hypoxide, given the size of drives these days loosing two drives is becoming far more likely. See Why RAID 5 stops working in 2009 - (http://blogs.zdnet.com/storage/?p=162) – Zoredache Jan 19 '10 at 20:33
  • @joeqwerty, fortunately the on-site server does have a recent version of the database from which I'm able to restore. However, over the last month I've been developing on our local mirror and that was the only place my work resided. It looks like I'll just have to rewrite the SQL. At least I'll have an opportunity to refactor it. – Daniel Jan 19 '10 at 20:43
  • Some type of distributed source control management tool might be useful as well. – Wayne Sheppard Jan 19 '10 at 21:00
  • This isn't the only recourse. As mentioned in the question, recovery specialists can rebuild RAID from multiple failed drives. My guess is that you are looking well over $3000 for the rebuild though. – Doug Luxem Jan 19 '10 at 21:36
  • @DLux: That's what I'm afraid of. We do have a contract with a government-grade data recovery specialist and we've dealt with them in the past, but they are extremely costly and don't promise anything. It's likely less expensive for me to do a rewrite than to send the drives in. – Daniel Jan 19 '10 at 22:28
  • @Wayne Sheppard: importing the latest schema into team foundation server presently. – Daniel Jan 19 '10 at 22:30
5

Restore from backup...

Wayne Sheppard
  • 422
  • 6
  • 10
  • Yep. One of the reasons that RAID-5 isn't encouraged anymore. 2-parity raid schemes are more reliable, and less likely to fail during recovery. – Matt Simmons Jan 19 '10 at 19:38
2

Go to the bank and get your backup tapes because it's gonna be a long night. Also, ask for RAID6 with a hot spare next time.

Wesley
  • 32,690
  • 9
  • 82
  • 117
1

This has happened to me twice, now I use RAID-6. This is common when after one drive fails, you attempt a restore and another one fails. Your best bet is to attempt a recovery of a bad drive to a blank one of equal or greater size then force it into the array. To restore to a blank drive just do a 'dd if=/dev/baddrive of=/dev/gooddrive bs=1024k conv=noerror,sync'. If there's a bad spot on the drive it will be replaced with zeros, but at least you get 99.99% of your data back.

0

If the drives have not physically failed and were just kicked out of the RAID set you may be able to recover the data. I have had luck recovering data in this situation by using a piece of software called R-Studio (r-studio.com). If the RAID controller in question has a pass-through mode turn this on and let R-Studio scan the drives. It will attempt to rebuild the RAID set and let you restore any non-corrupt data to another drive.

heydmj
  • 31
  • 1
  • Unfortunately the drives failed physically, and make a horrible clicking noise when attempting to boot. – Daniel Jan 19 '10 at 20:46