Edit 1: Just to clarify no data was written to the new extents. I know this because the admin grew the file system with fdisk right after doing the lvextend, he then rebooted for the changes to take affect. Upon reboot the new block device was lost.
Here is what the sys admin did.
A new block device was added, he then added the new block device to an existing volume group, and then he decided to do a lvextend on the volume group:
vgextend archive1-retain /dev/sdd1
lvextend -l +31 /dev/archive1-retain/data
Essentially extending the volume group with the new physical volume. He then rebooted and somehow lost the new physical volume member (sdd1). However the logical volume mounts and all of the data looks intact.
I am positive nothing actually wrote any data to the volume group after the second logical extent (sdd1) was accidentally added. Here is what I am left with, notice the unknown (missing) device on the second logical extent, which is to be expected:
archive1:~ # lvdisplay -am
/dev/archive1-retain/data: read failed after 0 of 512 at 4400059711488: Input/output error
/dev/archive1-retain/data: read failed after 0 of 512 at 4400059768832: Input/output error
Couldn't find device with uuid W0btGW-Zp3R-yoAC-dkow-PabM-1Vri-3VUQo1.
--- Logical volume ---
LV Name /dev/archive1-retain/data
VG Name archive1-retain
LV UUID fLG6tU-6tla-h49c-KOf2-LIAh-uIFy-wH0XB5
LV Write Access read/write
LV Creation host, time archive1, 2015-06-12 11:50:30 -0700
LV Status available
# open 1
LV Size 4.00 TiB
Current LE 65566
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 253:3
--- Segments ---
Logical extent 0 to 65534:
Type linear
Physical volume /dev/sda
Physical extents 0 to 65534
Logical extent 65535 to 65565:
Type linear
Physical volume unknown device
Physical extents 0 to 30
Here is what I tried. Looking back I shouldn't have used the --force, but luckily the volume was mounted at the time:
archive1:~ # vgreduce archive1-retain --removemissing
/dev/archive1-retain/data: read failed after 0 of 512 at 4400059711488: Input/output error
/dev/archive1-retain/data: read failed after 0 of 512 at 4400059768832: Input/output error
Couldn't find device with uuid W0btGW-Zp3R-yoAC-dkow-PabM-1Vri-3VUQo1.
WARNING: Partial LV data needs to be repaired or removed.
There are still partial LVs in VG archive1-retain.
To remove them unconditionally use: vgreduce --removemissing --force.
Proceeding to remove empty missing PVs.
archive1:~ # vgreduce archive1-retain --removemissing --force
/dev/archive1-retain/data: read failed after 0 of 512 at 4400059711488: Input/output error
/dev/archive1-retain/data: read failed after 0 of 512 at 4400059768832: Input/output error
Couldn't find device with uuid W0btGW-Zp3R-yoAC-dkow-PabM-1Vri-3VUQo1.
Removing partial LV data.
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
device-mapper: remove ioctl on failed: Device or resource busy
Unable to deactivate archive1--retain-data (253:3)
Unable to deactivate logical volume "data"
Some additional info:
archive1:~ # lsblk
NAME MAJ:MIN RM SIZE RO MOUNTPOINT
xvdb 202:16 0 249G 0
└─xvdb1 202:17 0 249G 0 /var/log/retain-tomcat7
xvda 202:0 0 80G 0
├─xvda1 202:1 0 2G 0 [SWAP]
└─xvda2 202:2 0 78G 0 /
sda 8:0 0 4T 0
└─archive1--retain-data (dm-3) 253:3 0 4T 0 /var/opt/beginfinite/retain
sdc 8:32 0 750G 0
└─gwsystem-data (dm-0) 253:0 0 750G 0 /gwsystem
sdb 8:16 0 1T 0
└─archive1--sql-data (dm-1) 253:1 0 1024G 0 /var/lib/mysql
archive1--retain-data-missing_1_0 (dm-2) 253:2 0 2G 0
└─archive1--retain-data (dm-3) 253:3 0 4T 0 /var/opt/beginfinite/retain
The logical volume mounts and is usable:
archive1:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 77G 35G 41G 46% /
udev 19G 120K 19G 1% /dev
tmpfs 19G 0 19G 0% /dev/shm
/dev/xvdb1 246G 62G 172G 27% /var/log/retain-tomcat7
/dev/mapper/gwsystem-data 739G 590G 112G 85% /gwsystem
/dev/mapper/archive1--sql-data 1.0T 330G 695G 33% /var/lib/mysql
/dev/mapper/archive1--retain-data 4.0T 2.9T 1.2T 73% /var/opt/beginfinite/retain
