My zfs pool is UNAVAIL and cannot be imported. I wonder if there would be any possibility that it is recovered or I can take out any data from it?
Background
I took over management of a server that has a Ubuntu 18.04 VM on VMware ESXi, where the zfs pool is, with two other VMs.
Since the zfs pool was full, I did zfs add to add extra 12 TB disc (sdi). (Given that other discs are 4 TB, this asymmetry is probably not ideal, but the disc seemed to be recognized and the VM was running healthy.) I think it was a problem that I set its disk mode to "Independent-persistent", which seems to mean the disc is not restored by snapshot.
Then, since a boot drive of the VM (sda, lvm, nextcloud-vg) was full, I tried to expand the lvm to sdj by lvextend. (I was too naive and stupid to take a snapshot before that.) Then, the VM got unbootable, where Grub did not find the boot disc. I noticed that this was because sdj was SCSI(0:10) and not recognized by BIOS at booting.
When I changed the SCSI ID of previous sdj to (0:2), Grub found the boot disc, but the VM was still unbootable since the Ubuntu files were somehow massively lost. Then, I restored the files in sda/nextcloud-vg by copying from old snapshot. Now, the VM is bootable but cannot read and import the zfs pool.
I really appreciate any comment and suggestion. Thank you so much for reading.
Healthy zfs pool as of Nov. 2019, which is accessible by a snapshot of VMware.
root@nextcloud:~# zpool status
pool: ncdata
state: ONLINE
scan: scrub repaired 0B in 51h42m with 0 errors on Tue Nov 12 04:06:08 2019
config:
NAME STATE READ WRITE CKSUM
ncdata ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
errors: No known data errors
root@nextcloud:~# zdb
ncdata:
version: 5000
name: 'ncdata'
state: 0
txg: 6813984
pool_guid: 4115695041733501295
errata: 0
hostname: 'nextcloud'
com.delphix:has_per_vdev_zaps
vdev_children: 7
vdev_tree:
type: 'root'
id: 0
guid: 4115695041733501295
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 12100899330530807388
path: '/dev/sdb1'
whole_disk: 1
metaslab_array: 131
metaslab_shift: 28
ashift: 12
asize: 42934468608
is_log: 0
create_txg: 4
com.delphix:vdev_zap_leaf: 129
com.delphix:vdev_zap_top: 130
children[1]:
type: 'disk'
id: 1
guid: 12620057802260459555
path: '/dev/sdc1'
whole_disk: 1
metaslab_array: 256
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1469
com.delphix:vdev_zap_leaf: 135
com.delphix:vdev_zap_top: 136
children[2]:
type: 'disk'
id: 2
guid: 3074381589662896744
path: '/dev/sdd1'
whole_disk: 1
metaslab_array: 384
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1472
com.delphix:vdev_zap_leaf: 137
com.delphix:vdev_zap_top: 138
children[3]:
type: 'disk'
id: 3
guid: 1759578235837374106
path: '/dev/sde1'
whole_disk: 1
metaslab_array: 140
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1475
com.delphix:vdev_zap_leaf: 512
com.delphix:vdev_zap_top: 513
children[4]:
type: 'disk'
id: 4
guid: 10518026248187884540
path: '/dev/sdf1'
whole_disk: 1
metaslab_array: 259
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1478
com.delphix:vdev_zap_leaf: 257
com.delphix:vdev_zap_top: 258
children[5]:
type: 'disk'
id: 5
guid: 8733140335356986716
path: '/dev/sdg1'
whole_disk: 1
metaslab_array: 640
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1481
com.delphix:vdev_zap_leaf: 386
com.delphix:vdev_zap_top: 387
children[6]:
type: 'disk'
id: 6
guid: 4551718362642717333
path: '/dev/sdh1'
whole_disk: 1
metaslab_array: 369
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1603266
com.delphix:vdev_zap_leaf: 349
com.delphix:vdev_zap_top: 368
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
Current situation.
root@nextcloud:~# zpool import
pool: ncdata
id: 4115695041733501295
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
config:
ncdata UNAVAIL insufficient replicas
sdb ONLINE
sde ONLINE
sdf ONLINE
sdg ONLINE
sdh ONLINE
sdi ONLINE
sdj ONLINE
sdi UNAVAIL corrupted data
root@nextcloud:~# zdb -C
ncdata:
version: 5000
name: 'ncdata'
state: 0
txg: 6813824
pool_guid: 4115695041733501295
errata: 0
hostname: 'nextcloud'
com.delphix:has_per_vdev_zaps
vdev_children: 7
vdev_tree:
type: 'root'
id: 0
guid: 4115695041733501295
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 12100899330530807388
path: '/dev/sdb1'
whole_disk: 1
metaslab_array: 131
metaslab_shift: 28
ashift: 12
asize: 42934468608
is_log: 0
create_txg: 4
com.delphix:vdev_zap_leaf: 129
com.delphix:vdev_zap_top: 130
children[1]:
type: 'disk'
id: 1
guid: 12620057802260459555
path: '/dev/sdc1'
whole_disk: 1
metaslab_array: 256
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1469
com.delphix:vdev_zap_leaf: 135
com.delphix:vdev_zap_top: 136
children[2]:
type: 'disk'
id: 2
guid: 3074381589662896744
path: '/dev/sdd1'
whole_disk: 1
metaslab_array: 384
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1472
com.delphix:vdev_zap_leaf: 137
com.delphix:vdev_zap_top: 138
children[3]:
type: 'disk'
id: 3
guid: 1759578235837374106
path: '/dev/sde1'
whole_disk: 1
metaslab_array: 140
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1475
com.delphix:vdev_zap_leaf: 512
com.delphix:vdev_zap_top: 513
children[4]:
type: 'disk'
id: 4
guid: 10518026248187884540
path: '/dev/sdf1'
whole_disk: 1
metaslab_array: 259
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1478
com.delphix:vdev_zap_leaf: 257
com.delphix:vdev_zap_top: 258
children[5]:
type: 'disk'
id: 5
guid: 8733140335356986716
path: '/dev/sdg1'
whole_disk: 1
metaslab_array: 640
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1481
com.delphix:vdev_zap_leaf: 386
com.delphix:vdev_zap_top: 387
children[6]:
type: 'disk'
id: 6
guid: 4551718362642717333
path: '/dev/sdh1'
whole_disk: 1
metaslab_array: 369
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
create_txg: 1603266
com.delphix:vdev_zap_leaf: 349
com.delphix:vdev_zap_top: 368
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
root@nextcloud:/dev/disk# zdb -d ncdata
zdb: can't open 'ncdata': Value too large for defined data type
root@nextcloud:/dev/disk# zdb -e -p /dev -d ncdata
zdb: can't open 'ncdata': File exists
root@nextcloud:/mytmpdev# zdb -e -p /dev -AAA -d ncdata
zdb: can't open 'ncdata': File exists
root@nextcloud:/dev/disk# zdb -e ncdata
Configuration for import:
version: 5000
pool_guid: 4115695041733501295
name: 'ncdata'
state: 0
vdev_children: 8
vdev_tree:
type: 'root'
id: 0
guid: 4115695041733501295
children[0]:
type: 'disk'
id: 0
guid: 12100899330530807388
whole_disk: 1
metaslab_array: 131
metaslab_shift: 28
ashift: 12
asize: 42934468608
is_log: 0
DTL: 1102
create_txg: 4
path: '/dev/sdb1'
children[1]:
type: 'disk'
id: 1
guid: 12620057802260459555
whole_disk: 1
metaslab_array: 256
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
DTL: 1109
create_txg: 1469
path: '/dev/sde1'
children[2]:
type: 'disk'
id: 2
guid: 3074381589662896744
whole_disk: 1
metaslab_array: 384
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
DTL: 1108
create_txg: 1472
path: '/dev/sdf1'
children[3]:
type: 'disk'
id: 3
guid: 1759578235837374106
whole_disk: 1
metaslab_array: 140
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
DTL: 1107
create_txg: 1475
path: '/dev/sdg1'
children[4]:
type: 'disk'
id: 4
guid: 10518026248187884540
whole_disk: 1
metaslab_array: 259
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
DTL: 1103
create_txg: 1478
path: '/dev/sdh1'
children[5]:
type: 'disk'
id: 5
guid: 8733140335356986716
whole_disk: 1
metaslab_array: 640
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
DTL: 1106
create_txg: 1481
path: '/dev/sdi1'
children[6]:
type: 'disk'
id: 6
guid: 4551718362642717333
whole_disk: 1
metaslab_array: 369
metaslab_shift: 35
ashift: 12
asize: 4398031306752
is_log: 0
DTL: 1105
create_txg: 1603266
path: '/dev/sdj1'
children[7]:
type: 'disk'
id: 7
guid: 5107684954562511391
whole_disk: 1
metaslab_array: 1058
metaslab_shift: 36
ashift: 12
asize: 13194124328960
is_log: 0
create_txg: 6813878
path: '/dev/sdk1'
rewind-policy:
rewind-request-txg: 18446744073709551615
rewind-request: 2
zdb: can't open 'ncdata': File exists
ZFS_DBGMSG(zdb):
root@nextcloud:/mytmpdev# zdb -l /dev/sdb1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'ncdata'
state: 0
txg: 9549126
pool_guid: 4115695041733501295
errata: 0
hostname: 'nextcloud'
top_guid: 12100899330530807388
guid: 12100899330530807388
vdev_children: 8
vdev_tree:
type: 'disk'
id: 0
guid: 12100899330530807388
path: '/dev/sdb1'
whole_disk: 1
metaslab_array: 131
metaslab_shift: 28
ashift: 12
asize: 42934468608
is_log: 0
DTL: 1102
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3
root@nextcloud:/mytmpdev# zdb -l /dev/sdk1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'ncdata'
state: 0
txg: 6813880
pool_guid: 4115695041733501295
errata: 0
hostname: 'nextcloud'
top_guid: 5107684954562511391
guid: 5107684954562511391
vdev_children: 8
vdev_tree:
type: 'disk'
id: 7
guid: 5107684954562511391
path: '/dev/sdi1'
whole_disk: 1
metaslab_array: 1058
metaslab_shift: 36
ashift: 12
asize: 13194124328960
is_log: 0
create_txg: 6813878
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3
root@nextcloud:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
└─sda1 8:1 0 40G 0 part
├─nextcloud--vg-root 253:0 0 1T 0 lvm /
└─nextcloud--vg-swap_1 253:1 0 976M 0 lvm [SWAP]
sdb 8:16 0 40G 0 disk
├─sdb1 8:17 0 40G 0 part
└─sdb9 8:25 0 8M 0 part
sdc 8:32 0 4T 0 disk
└─nextcloud--vg-root 253:0 0 1T 0 lvm /
sdd 8:48 0 4T 0 disk
├─sdd1 8:49 0 1M 0 part
├─sdd2 8:50 0 976.8G 0 part
└─sdd3 8:51 0 976.8G 0 part
sde 8:64 0 4T 0 disk
├─sde1 8:65 0 4T 0 part
└─sde9 8:73 0 8M 0 part
sdf 8:80 0 4T 0 disk
├─sdf1 8:81 0 4T 0 part
└─sdf9 8:89 0 8M 0 part
sdg 8:96 0 4T 0 disk
├─sdg1 8:97 0 4T 0 part
└─sdg9 8:105 0 8M 0 part
sdh 8:112 0 4T 0 disk
├─sdh1 8:113 0 4T 0 part
└─sdh9 8:121 0 8M 0 part
sdi 8:128 0 4T 0 disk
├─sdi1 8:129 0 4T 0 part
└─sdi9 8:137 0 8M 0 part
sdj 8:144 0 4T 0 disk
├─sdj1 8:145 0 4T 0 part
└─sdj9 8:153 0 8M 0 part
sdk 8:160 0 12T 0 disk
├─sdk1 8:161 0 12T 0 part
└─sdk9 8:169 0 8M 0 part
sr0 11:0 1 1024M 0 rom
root@nextcloud:/dev/disk# parted -l
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 42.9GB 42.9GB primary boot, lvm
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 42.9GB 42.9GB zfs zfs-7c215dc258f390a2
9 42.9GB 42.9GB 8389kB
Error: /dev/sdc: unrecognised disk label
Model: VMware Virtual disk (scsi)
Disk /dev/sdc: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
Model: VMware Virtual disk (scsi)
Disk /dev/sdd: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB bios_grub
2 2097kB 1049GB 1049GB ext4
3 1049GB 2098GB 1049GB ext4
Model: VMware Virtual disk (scsi)
Disk /dev/sde: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 4398GB 4398GB zfs zfs-bb77ffc8c56c704c
9 4398GB 4398GB 8389kB
Model: VMware Virtual disk (scsi)
Disk /dev/sdf: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 4398GB 4398GB zfs zfs-e5f9cec454077158
9 4398GB 4398GB 8389kB
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/nextcloud--vg-swap_1: 1023MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 1023MB 1023MB linux-swap(v1)
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/nextcloud--vg-root: 1100GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 1100GB 1100GB ext4
Model: VMware Virtual disk (scsi)
Disk /dev/sdk: 13.2TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 13.2TB 13.2TB zfs zfs-e5495635496c5b84
9 13.2TB 13.2TB 8389kB
Model: VMware Virtual disk (scsi)
Disk /dev/sdi: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 4398GB 4398GB zfs zfs-b18c76425dec00d4
9 4398GB 4398GB 8389kB
Model: VMware Virtual disk (scsi)
Disk /dev/sdg: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 4398GB 4398GB zfs zfs-7662c1d580329ece
9 4398GB 4398GB 8389kB
Model: VMware Virtual disk (scsi)
Disk /dev/sdj: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 4398GB 4398GB zfs zfs-45572a27b3ca3dd5
9 4398GB 4398GB 8389kB
Model: VMware Virtual disk (scsi)
Disk /dev/sdh: 4398GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 4398GB 4398GB zfs zfs-a3d1bd676c04ef35
9 4398GB 4398GB 8389kB
