How to repair zsysctl after manually deleting snapshots with `zfs destroy`?

Question

I am running Linux Mint 20.3 with ZFS (including for /). Up until very recently, whenever I installed or removed packages, it would also create a snapshot so that I could select it when booting to revert a change that might have broken the system.

But it seems there is a limitation of this system, where it can't cope with a large (1200+) number of snapshots. I can't find the issue anymore, but it suggested running for dataset in $(zfs list -t snapshot -o name | grep @autozsys); do sudo zfs destroy "$dataset"; done to delete most snapshots.

Since I didn't need these older snapshots, I gave it a go. I couldn't delete some because they were clones of others and not knowing whether this would make me lose live data, I left it at that.

I later learned about the zsysctl command to manage snapshots... And it seems I broke my system by not using that command but zfs destroy directly.

Now, whenever I apt install or apt remove, I'm getting this error (after a 30s pause): ERROR Service took too long to respond. Disconnecting client.

I think zsysctl is partially broken because running zsysctl show results in the last entry in the lists for bpool and rpool having a date of 0001-01-01 00:00:00 which is obviously wrong:

$ zsysctl show
Name:           rpool/ROOT/ubuntu_uyv8gh
ZSys:           true
Last Used:      current
History:        
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_nma9lm
    Created on: 2022-11-17 17:43:38
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_1ayzvs
    Created on: 2022-11-14 13:22:37
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_yv8nrd
    Created on: 2022-11-11 17:13:10
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_yu0btn
    Created on: 2022-02-15 15:58:44
  - Name:       rpool/ROOT/ubuntu_6xf31g
    Created on: 2021-10-22 17:13:50
Users:
  - Name:    pierre
    History: 
     - rpool/USERDATA/pierre_gg46xm@autozsys_nma9lm (2022-11-17 17:43:40)
     - rpool/USERDATA/pierre_gg46xm@autozsys_m6wubi (2022-11-17 14:29:58)
     - rpool/USERDATA/pierre_gg46xm@autozsys_17okng (2022-11-17 11:29:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_dgvgfz (2022-11-17 10:29:19)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5mk4hd (2022-11-16 15:55:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_z90d2e (2022-11-16 14:54:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_pfutl5 (2022-11-16 13:54:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5dfwde (2022-11-16 12:42:58)
     - rpool/USERDATA/pierre_gg46xm@autozsys_iuusno (2022-11-16 11:42:54)
     - rpool/USERDATA/pierre_gg46xm@autozsys_zzrd7w (2022-11-16 10:41:57)
     - rpool/USERDATA/pierre_gg46xm@autozsys_d4kkrw (2022-11-16 09:41:55)
     - rpool/USERDATA/pierre_gg46xm@autozsys_2ws9fj (2022-11-15 16:06:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_cfb7iu (2022-11-15 15:06:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_3rcqfi (2022-11-15 13:35:17)
     - rpool/USERDATA/pierre_gg46xm@autozsys_56mb7s (2022-11-15 12:07:32)
     - rpool/USERDATA/pierre_gg46xm@autozsys_9b068o (2022-11-15 11:06:38)
     - rpool/USERDATA/pierre_gg46xm@autozsys_o610qz (2022-11-15 10:06:35)
     - rpool/USERDATA/pierre_gg46xm@autozsys_fw4jbj (2022-11-15 08:38:21)
     - rpool/USERDATA/pierre_gg46xm@autozsys_j9ow55 (2022-11-14 17:00:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_mfc8hk (2022-11-14 15:59:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_1ayzvs (2022-11-14 13:22:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_y6xypr (2022-11-14 12:57:47)
     - rpool/USERDATA/pierre_gg46xm@autozsys_o0aj49 (2022-11-14 11:57:46)
     - rpool/USERDATA/pierre_gg46xm@autozsys_iljt6z (2022-11-14 10:57:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_2fmmg5 (2022-11-14 09:56:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_lxkohx (2022-11-14 08:55:51)
     - rpool/USERDATA/pierre_gg46xm@autozsys_czv8zx (2022-11-13 16:58:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_8kl96y (2022-11-13 15:57:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5ks5h8 (2022-11-13 14:56:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5m7wwk (2022-11-13 13:56:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_40udn4 (2022-11-13 12:28:40)
     - rpool/USERDATA/pierre_gg46xm@autozsys_uagprp (2022-11-13 11:28:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_4vbncy (2022-11-13 10:28:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_9buhaa (2022-11-13 09:26:59)
     - rpool/USERDATA/pierre_gg46xm@autozsys_92de9m (2022-11-12 14:48:45)
     - rpool/USERDATA/pierre_gg46xm@autozsys_x5vgcm (2022-11-12 12:00:12)
     - rpool/USERDATA/pierre_gg46xm@autozsys_pm05qo (2022-11-12 10:43:35)
     - rpool/USERDATA/pierre_gg46xm@autozsys_kxd0z1 (2022-11-12 09:43:32)
     - rpool/USERDATA/pierre_gg46xm@autozsys_welapl (2022-11-11 17:18:27)
     - rpool/USERDATA/pierre_gg46xm@autozsys_yv8nrd (2022-11-11 17:13:11)
     - rpool/USERDATA/pierre_gg46xm-rpool.ROOT.ubuntu-6xf31g (2022-11-09 07:17:16)
     - rpool/USERDATA/pierre_gg46xm@autozsys_yu0btn (2022-02-15 15:58:45)
     - rpool/USERDATA/pierre_hntkn5 (0001-01-01 00:00:00)
  - Name:    root
    History: 
     - rpool/USERDATA/root_gg46xm@autozsys_nma9lm (2022-11-17 17:43:40)
     - rpool/USERDATA/root_gg46xm@autozsys_1ayzvs (2022-11-14 13:22:39)
     - rpool/USERDATA/root_gg46xm@autozsys_yv8nrd (2022-11-11 17:13:11)
     - rpool/USERDATA/root_gg46xm-rpool.ROOT.ubuntu-6xf31g (2022-11-09 07:17:16)
     - rpool/USERDATA/root_gg46xm@autozsys_yu0btn (2022-02-15 15:58:45)
     - rpool/USERDATA/root_hntkn5 (0001-01-01 00:00:00)

Running zsysctl save -vv doesn't give much more information:

$ zsysctl save -vv
DEBUG /zsys.Zsys/SaveUserState() call logged as [e560ff96:3c3c6ace] 
DEBUG Check if grpc request peer is authorized     
DEBUG Polkit call result, authorized: true         
DEBUG Didn't receive any information from service in 30s 
ERROR Service took too long to respond. Disconnecting client.

How do I recover from this partially broken state?

ewwhite · Answer 1 · 2022-11-28T15:16:11.927

1

You can remove multiple snapshots by using %.

Example:

zfs destroy rpool/USERDATA/pierre_gg46xm@autozsys_nma9lm%autozsys_yv8nrd

This removes everything from autozsys_nma9lm through autozsys_yv8nrd.

If you have clones intermixed amongst the 1200+ snapshots, then it may be worth reevaluating the setup or separating your data into data-specific pools.

I see you've used the zsysctl. I'm not familiar with it, but ZFS is already streamlined with two main commands; zfs and zpool.

There doesn't seem to be much utility to using zsysctl here because it's definitely complicated your setup. It may be an unnecessary abstraction and doesn't have a future on the platform. Consider removing it.

zsys being removed from ubuntu #235

zsys cannot handle large amounts of snapshots #224

(I'd say the same for ZFS root and the high number of snapshots)

edited Nov 28 '22 at 15:16

answered Nov 27 '22 at 20:57

ewwhite

197,159
92
443
809

Thanks. I have two issues with this: 1. I have clones, and I don't know how to make sure I'm not deleting live data by deleting the clones; 2. I don't know if I can just yank zsys like that and what to replace it with as having the snapshots in grub is nice. zfsbootmenu is mentioned, but I'm wary of relying on my making it work with the system rather than having an "out of the box" solution. – Pierre Nov 28 '22 at 18:42
Is this a server or professional environment? – ewwhite Nov 28 '22 at 19:10
No, it’s my workstation. – Pierre Dec 03 '22 at 07:33

How to repair zsysctl after manually deleting snapshots with `zfs destroy`?

1 Answers1