5

Yesterday, I put a new disk into my server. Sadly, I didn't check the disk before of failures.

I added it to my pool with the command zpool add nas /dev/disk/by-id/scsi-SATA_ST31500341AS_9VS27Z4M-part1

Short after, the CPU-load of the server went nearly to infinity, I couldn't even relogin.

So I performed a hard-reboot (Alt + SysRq + b), but the server couldn't boot. (After the GRUB showed up, nothing more happened for about 5 minutes. Then, I shut it down and took out the new disk. I booted up and it worked.

But now, I have the problem, that I can't access the so-called "nas"-pool because the last (the new) disk shows as status "UNAVAIL", and because it's no mirrored pool, the whole pool is in state UNAVAIL.

If I put in the disk again and do a zpool online nas /dev/disk/by-id/scsi-SATA_ST31500341AS_9VS27Z4M-part1 it doesn't work and tells me "the disk could not be found".

So i tried some possibilities I read from the oracle-docs and I exported it with zpool export nas. Now, I'm not even able to import the pool.

zpool import nas -f
cannot import 'nas': one or more devices is currently unavailable

And if I look at zpool import, it tells me:

pool: nas
     id: 3366469163144781663
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://zfsonlinux.org/msg/ZFS-8000-6X
 config:

        nas                                               UNAVAIL  missing device
          dm-name-linuxServer-nas                         ONLINE
          ata-WDC_WD20EARX-00PASB0_WD-WCAZAC521840-part1  ONLINE
          ata-WDC_WD20EFRX-68AX9N0_WD-WMC300228535-part1  ONLINE

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.

To be clear: the pool is completely irrelevant, the data is not. If I'd be able to access the data somehow, I could export it to a external HDD.

Yes, I have no backup (shame on me!) but I don't have usually another 6TB for the backup somewhere.

Is there any possibility to access this data? Maybe faking the disk so that zpool thinks, it's available or something like this?

Any help would greatly be appreciated.

ewwhite
  • 197,159
  • 92
  • 443
  • 809
UeliDeSchwert
  • 152
  • 7
  • So what happened to the disk? – Michael Hampton Apr 19 '14 at 22:09
  • Nothing happened. I got this disk from a friend of mine. First of all, I made a new partition-table with only one partition and added this partition to the pool. After I did this, something went wrong. I think the disk was faulted even as I put it in there, but somehow I've been able to alter the partition table - no I can't even open it via fdisk: Input/Output-Error. – UeliDeSchwert Apr 19 '14 at 22:11

2 Answers2

5

I hate to say this, but do you know that you were basically adding unraided disks to your pool?
The command you provided basically says, "Add another disk to pool nas and stripe it with the existing disks."

Is that what you meant to do?

The pool is done at this point, especially if anything was written to the bad disk. If this were a pair of mirrors, the pool would be in state: FAULTED and you'd be able to recover. As-is, you've probably lost your data.

ewwhite
  • 197,159
  • 92
  • 443
  • 809
  • I didn't put any new data to it, so I don't think that something new was written to the disk (except of some basical metadata, I don't know how zfs acts exactly). So is there any possibility to recover the data (which is 100% on the other three disks)? – UeliDeSchwert Apr 19 '14 at 22:14
  • @stueliueli Other *three* disks? How many did you start with? – ewwhite Apr 19 '14 at 22:15
  • I had three disks for an extended period of time running. Now I just added a new one with the "zpool add" command... – UeliDeSchwert Apr 19 '14 at 22:17
  • 1
    @stueliueli So you had three disks running without RAID, and added a fourth without RAID. The effect is the same. The pool is `UNAVAILABLE`. There may be some ZFS `zdb` magic that could help here. That is beyond my skill set, though. – ewwhite Apr 19 '14 at 22:19
  • Hmm thanks for your help. Maybe it may be lost (The essential parts I got backups). But if there's the slightest possibility that I could recover some data, this would be very interesting. If I don't find an solution to this somewhere else, I will mark your answer as correct. But if there's some zdb-magic, I'll try this. – UeliDeSchwert Apr 19 '14 at 22:21
  • @stueliueli If your data really is that important, send all the disks including the defective one to a professional data recovery company. And of course, never do this again. – Michael Hampton Apr 19 '14 at 22:30
  • @MichaelHampton There may be something that can be done... but this was a mess to start with. – ewwhite Apr 19 '14 at 22:34
  • @Michael Hampton no, it isn't thaat important ;) And yes, I have learned my lesson ;) All data is somewhere available (it was more like a save for me, where I put all the data I downloaded from somewhere (because my connection is extremely slow...). But thank you for the help :) – UeliDeSchwert Apr 19 '14 at 22:36
  • @ewwhite hmm, what do you think of? From now, I think like I lost my data, so a mess is no problem. Nothing chan go wrong ;) – UeliDeSchwert Apr 19 '14 at 22:40
  • @stueliueli I've asked an expert to look at this. – ewwhite Apr 19 '14 at 22:40
  • @stueliueli Try `zpool import -D nas` – ewwhite Apr 19 '14 at 22:42
  • It tells me that `cannot import 'nas': no such pool available` but with only `zpool import` I see the nas-pool. According to the zpool-manpage, -D imports destroyed pools. Do I have to destroy the pool first? – UeliDeSchwert Apr 19 '14 at 22:49
2

My suggestion to you would be to get a duplicate set of disks and ghost the raw contents over to the other set of disks for starters. Once you get the data restored if possible you then have the disks to make a mirror or other backup on. It will allow you much more leeway in trying to recover your pool that way. you are also leaving out important information like what mode you created the pool in. z-1 z-2

see http://www.joyent.com/blog/zfs-forensics-recovering-files-from-a-destroyed-zpool you may have some luck with that. I would also verify that the disk really is bad as well. If it isn't then you may have another issue going on like a flaky controller. if the disk is bad you may be able to deal with it by ghosting contents. I haven't messed with zfs importing disks enough to know if it is possible to tell zfs that the replacement disk is the failed one, especially since it is by id and not slot. It may not be possible to trick zfs in to using the ghosted data on a replacement drive to recover from the fault. That thought though is what comes to mind from my other experiances in computers.

Kendrick
  • 293
  • 4
  • 16