Recover from "FAULTED" zfs pool. "-F" option seems to be ignored

Question

On Ubuntu, I recently started trying out ZFS on an external USB hard drive. I now seem to have a corrupted zfs pool, which probably happened simply as a result of me shutting down my computer (with poweroff).

$ zpool import
   pool: zfs
     id: 1234512345123451234
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:

        zfs                      FAULTED  corrupted data
          usb-Disk_Name_etc-0:0  ONLINE

When trying to import it I get this helpful message explaining that I'll lose about 1 second of data (which would be fine) and that I can attempt recovery with the "-F" option:

$ zpool import zfs
cannot import 'zfs': I/O error
    Recovery is possible, but will result in some data loss.
    Returning the pool to its state as of Thu 12 Mar 2020 00:58:03 GMT
    should correct the problem.  Approximately 1 seconds of data
    must be discarded, irreversibly.  Recovery can be attempted
    by executing 'zpool import -F zfs'.  A scrub of the pool
    is strongly recommended after recovery.

Using the dry-run flag, I get confirmation that -F should work:

$ zpool import -Fn zfs
Would be able to return zfs to its state as of Thu 12 Mar 2020 00:58:03 GMT.
Would discard approximately 1 seconds of transactions.

But, when I try the real thing, it seems to completely ignore the "-F" option and repeats the original message:

$ zpool import -F zfs
cannot import 'zfs': I/O error
    Recovery is possible, but will result in some data loss.
    Returning the pool to its state as of Thu 12 Mar 2020 00:58:03 GMT
    should correct the problem.  Approximately 1 seconds of data
    must be discarded, irreversibly.  Recovery can be attempted
    by executing 'zpool import -F zfs'.  A scrub of the pool
    is strongly recommended after recovery.

I found mention of the "-X" option from this answer: https://serverfault.com/a/645866/74394 but I continue to get exactly the same message. I've tried all of these option combinations, both with the pool name (zfs) and the numerical ID number (like 1234512345123451234), and I always get exactly the same message as above suggesting to use the -F option.

-F -FX -f -fF -fFX

Am I doing something obviously wrong here? ZFS seems incredibly flaky if I've irretrievably lost the whole disk just because of 1 second of data due to a system shutdown! I don't have a backup copy of the pool since I'd only just started trying out ZFS. Thanks for any help.

Gordan Bobić · Accepted Answer · 2020-03-14T21:21:25.543

2

If that USB device is lying about commits, all bets are off because the writes could be re-ordered, and between write re-ordering and lying about barriers, all bets are off on whether the metadata is trashed.

Try: zdb -e zfs -ul

This will hopefully list some txgs.

Then try: zpool import -f -FX -N -T [txg_number]

where you got the txg_number from zdb above.

Other things to try: Update ZFS to latest 0.8.3. What ships with Ubuntu is quite ancient and there have been a lot of fixes and improvements since that version.

edited Mar 14 '20 at 21:21

answered Mar 14 '20 at 17:16

Gordan Bobić

971
4
11

Thanks for this. `zdb -e zfs -ul` gives me 32 Uberblocks under "LABEL 0" and another one under "LABEL 2", although the txg of the latter matches with one of the txg's in the former list. I tried doing `zpool import -f -FX -N -T ` for every single txg listed, but sadly it gives me this message for every single one: `[...] status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag.` – Pryo Mar 14 '20 at 19:30
1

Try renaming your zfs cache file. It may also be worth trying upgrading to latest ZFS (0.8.3), and trying with that. The version that ships with Ubuntu is quite old. – Gordan Bobić Mar 14 '20 at 19:34
1

Oh my, you're a life saver. It was as simple as updating zfs. Do you want to maybe put an answer suggesting to update and I'll accept that as the solution? I built the latest 0.8.3 version from source and `zpool import -F zfs` just worked. – Pryo Mar 14 '20 at 20:24
1

Updated the original answer. – Gordan Bobić Mar 14 '20 at 21:22

Recover from "FAULTED" zfs pool. "-F" option seems to be ignored

1 Answers1