4

After running # yum update zfs and restarting my CentOS 6 ZFS on Linux system can no longer see or interact with it's zpools.

# zpool status
no pools available
# zpool import
zpool: ../../lib/libzfs/libzfs_import.c:356: Assertion `nvlist_lookup_uint64(zhp->zpool_config, ZPOOL_CONFIG_POOL_GUID, &theguid) == 0' failed.
Aborted
# dmesg | grep -i spl 
SPL: Loaded module v0.6.2-1 
SPL: using hostid 0x43a4c8a0

This is a canary system, so I'm not worried about its contents, but I have other machines that I would eventually like to upgrade safely. What did I do wrong, and more importantly how can I do it right for the important systems?

ewwhite
  • 197,159
  • 92
  • 443
  • 809
84104
  • 12,905
  • 6
  • 45
  • 76

3 Answers3

5

This came up in discussion on the ZFS mailing list.

The kernel ABI changed between 0.6.2 and 0.6.3 in such a way that 0.6.3 user-land tools can't interact with 0.6.2 kernel modules, which is the configuration after upgrading to 0.6.3. According to numerous messages on this list, the right approach is simply to reboot.

This is covered in the ZFS bug report: https://github.com/zfsonlinux/zfs/issues/2400#issuecomment-46118193

I only use ZFS for data partitions, so I uninstalled the packages, got rid of the weak-modules references and reinstalled ZFS.

# rpm -e `rpm -qa | egrep '(spl|zfs)'`
# cd /lib/modules/2.6.32-431.17.1.el6.x86_64/weak-updates/
# rm -fr spl* z*
# yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release$(rpm -E %dist).noarch.rpm
# yum install zfs

Edit: I still had to reboot.

Edit:

Following both workarounds presented resulted in some problems with the ZFS installation. There are some issues with module loading due to a critical missing parameter (zfs: Unknown parameterzfs_vdev_max_pending'`). I'd deem this release unsafe for general release or blind updates on existing systems.

ewwhite
  • 197,159
  • 92
  • 443
  • 809
5

That looks like...a bug. In fact, it is a bug. The ZFS userland is updated, but the kernel modules aren't being updated by DKMS. This assertion is caused by the version mismatch.

You can work around it by a process given in a comment to the bug, by removing the old kernel modules and then reinstalling everything.

$ find /lib/modules/$(uname -r)/extra -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f
$ find /lib/modules/$(uname -r)/weak-updates -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f
$ yum reinstall zfs-release
$ yum reinstall $(rpm -qa | egrep "zfs|spl")
Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
4

If you ever end up on this page and your running Debian, you only have to do the following to fix the issue:

# apt-get update
# apt-get dist-upgrade
# reboot

The issue occurred because a simple update doesn't not replace old library files with the newer ones. This is why a dist-upgrade is needed. From the apt-get manpage:

dist-upgrade in addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages; apt-get has a "smart" conflict resolution system, and it will attempt to upgrade the most important packages at the expense of less important ones if necessary. The dist-upgrade command may therefore remove some packages.

Source: http://bernaerts.dyndns.org/linux/75-debian/312-debian-wheezy-zfsonlinux-zpool-lost

DjDCH
  • 141
  • 3