Procedure/Effects of hot-removing a mirrored vdev disk

Question

I have two-ish questions here based off of a common setup. A quick disclaimer: I am using freenas and have not fully digested ZFS terminology and freenas butchers ZFS terminology in their UI anyway. I'll accept answers that use the terminal or the Freenas UI (bonus points for both ;)).

I have a single volume (zpool?) with 1 mirror vdev comprised of two 3TB disks.

1. What is the proper procedure to physically remove one of the disks and then place it back?

2. Probably unwisely, I removed one of the disks without doing any commands. Immediately I was alerted that the volume was DEGRADED (expected). This status persisted after I placed the drive back in (it did not seem to recognize the drive as the removed drive - or if it did I did not know how to reattach it). I rebooted the server and it now shows the volume as healthy but under Volume Manager->Volume Stats, there is a 182 in the checksum column of one of the drives but not the other (though I don't know if that was there beforehand).

   • How should I have handled the situation?
   • Can this cause data loss or should ZFS recover from this situation fine?
   • If it can cause data loss/corruption/whathaveyou, how do I check and recover from this?

Finally, additional bonus points for links to concise ZFS primers that aren't textbooks and don't delve into uselessly obscure parts of ZFs. :P

Answer 1

Disclaimer: I have not used FreeNAS personally, so some answers may not be the canonical way to do it - please consult the manual beforehand. That said, they are the way Oracle recommends it, so there should be some truth behind it.

---

1. It depends on your goal:
   • If you want to replace a faulted drive with a replacement, you do it in the FreeNAS Web UI, or from the console with zpool replace poolname olddevice newdevice.
   • If you want to remove a disk from a mirror vdev permanently (and reduce the mirror to basic, losing all redundancy in the process), you do it in the GUI or from the console with zpool detach poolname olddevice (use zpool attach poolname olddevice newdevice to re-attach).
   • If you want to temporarily take a device out of the active pool without removing or detaching it, use the GUI or from the console with zpool offline poolname olddevice. This is useful if troubleshooting larger arrays, I only write it for completeness here.
2. • Reboot exports all pools and then imports them. On import, ZFS checks if all disks are present. So instead of rebooting you could also do zpool export poolname follwed by zpool import poolname to achieve the same. It is possible that you have to configure the disk after insertion (this is the case on standard illumos systems) before importing the pool. If you do not want to take the pool down, you can also remove the faulted device and then replace it. This way you have no downtime, but resilvering. I would personally always choose the first solution, except on large arrays, because it is much safer.
   • See next question.
   • If the checksum was on one drive only, you are fine. Use zpool clear poolname to remove the error message. If the checksum errors were on both drives, it depends on the specific case. Normally ZFS would tell you if and what files are affected.

Bonus points answer:

• Oracle help and documentation pages, as linked above
• man zpool, man zfs and man /usr/bin/chmod for zpool, file systems and ACL configurations. They are also available online from Oracle and they are very very good. In fact, you almost need nothing else, because they also have plenty of real world examples (much better than GNU manpages).
• Solaris Best Practices Guide
• FreeNAS Worst practices (pitfalls to avoid)
• Nex7's Things Nobody Told You About ZFS
• Constantin Gonzalez' blog
• ZFS Cheat Sheet with most useful commands in compact form