LVM snapshot of btrfs volume changes mounted device

Question

I have a btrf volume on top of LVM. Now I want to do a snapshot on the lvm level (NOT on the btrfs level). But every time I create the LVM snapshot, btrfs changes the mounted block device to the snapshot like I was using some kind of --bind mount option.

Situation:

# mount | grep libvirt
/dev/dm-4 on /var/lib/libvirt/images type btrfs (rw,relatime,space_cache)
# ls -l /dev/mapper | grep dm-4
lrwxrwxrwx 1 root root       7 Mär 17 01:18 system-vm_disks -> ../dm-4
# lvcreate -s /dev/system/vm_disks -n vm_backup -L 32G
  Logical volume "vm_backup" created
# mount | grep libvirt
/dev/dm-5 on /var/lib/libvirt/images type btrfs (rw,relatime,space_cache)
# ls -l /dev/mapper | grep dm-5
lrwxrwxrwx 1 root root       7 Mär 17 01:18 system-vm_backup -> ../dm-5
# mount /dev/system/vm_backup /mnt/test
# touch /mnt/test/touchME
# ls -l /var/lib/libvirt/images/touchME
-rw-r--r-- 1 root root 0 Mär 17 01:26 /var/lib/libvirt/images/touchME

When I remove the snapshot:

# umount /mnt/test
# lvremove /dev/system/vm_backup 
Do you really want to remove active logical volume vm_backup? [y/n]: y
  Logical volume "vm_backup" successfully removed
# mount | grep libvirt
/dev/dm-4 on /var/lib/libvirt/images type btrfs (rw,relatime,space_cache)
# ls -l /dev/mapper | grep dm-4
lrwxrwxrwx 1 root root       7 Mär 17 01:21 system-vm_disks -> ../dm-4
# ls -l /var/lib/libvirt/images/touchME
-rw-r--r-- 1 root root 0 Mär 17 01:26 /var/lib/libvirt/images/touchME

I expect my snapshot to be a real snapshot not something like a --bind mount. I'm using the LVM snapshots to backup a consistent system state via rsync to our backup server. And I don't want to use btrfs snapshots for various reasons:

• I want to backup every subvolume and every btrfs snapshot inside the vm_disks LV (and I don't know how much and which snapshots/subvolumes exist)
• My backup strategy should not be filesystem dependant. Ideally it should not be neccessary to change anything else when changing the filesystem at /var/lib/libvirt/images

My System:

# uname -a
Linux laptop 3.12-1-amd64 #1 SMP Debian 3.12.9-1 (2014-02-01) x86_64 GNU/Linux
# lvm version
  LVM version:     2.02.104(2) (2013-11-13)
  Library version: 1.02.83 (2013-11-13)
  Driver version:  4.26.0
# btrfs --version
Btrfs v3.12

I have to use at least kernel 3.9 or newer since I use the IPv6 NAT features provided by Linux 3.9 or newer (yes, I know you should not use NAT with IPv6, but thats an other story).

Thanks for your help!

Edit:

I did some experiments using dd and loop devices. This behavior is not specific to LVM at all.

Tests:

# mkfs.btrfs /dev/loop0
[...]
# mount /dev/loop0 original
# touch original/original_file
# ls -l original
-rw-r--r-- 1 root root 0 Mar 28 21:42 original_file
# dd if=/dev/loop0 of=/dev/loop1
509312+0 records in
509312+0 records out
260767744 bytes (261 MB) copied, 1.76431 s, 148 MB/s
# mount /dev/loop1 clone
# touch clone/expected_clone_file
# ls -l clone
-rw-r--r-- 1 root root 0 Mar 28 21:44 expected_clone_file
-rw-r--r-- 1 root root 0 Mar 28 21:42 original_file
# ls -l original
-rw-r--r-- 1 root root 0 Mar 28 21:44 expected_clone_file
-rw-r--r-- 1 root root 0 Mar 28 21:42 original_file
# umount clone
# umount original
# mount /dev/loop1 clone
# ls -l clone
-rw-r--r-- 1 root root 0 Mar 28 21:42 original_file
# umount clone
# mount /dev/loop0 original
# ls -l original
-rw-r--r-- 1 root root 0 Mar 28 21:44 expected_clone_file
-rw-r--r-- 1 root root 0 Mar 28 21:42 original_file

So whenever you try to mount a new device with a cloned btrfs filesystem inside you end up using the old already mounted device (but nothing in the output of mount is properly indicating this, as you can see in the LVM experiment above). All requests thus end up using the old device. You are not able to mount the cloned fs until you unmount the original fs (and you cannot mount the original fs while the cloned one is mounted).

My question now is: How can I change the uuid of the cloned btrfs to some new unused uuid. After that I would be able to mount the cloned device alongside the original one, I suspect.

Answer 1

It seems that udev is causing this behavior.

Performing the lvcreate (or losetup) causes udev "change" actions on the "block" system:

# udevadm monitor
...
UDEV  [62084.032411] change   /devices/virtual/block/dm-7 (block)
UDEV  [62084.469374] change   /devices/virtual/block/dm-6 (block)
UDEV  [62084.582549] change   /devices/virtual/block/dm-6 (block)
UDEV  [62084.606191] change   /devices/virtual/block/dm-5 (block)
...

which triggers (in my case) the rules from

/lib/udev/rules.d/64-btrfs.rules

and calls the builtin udev command:

IMPORT{builtin}="btrfs ready $devnode"

which passes through src/udev/udev-builtin-btrfs.c:52

err = ioctl(fd, BTRFS_IOC_DEVICES_READY, &args);

To land in kernel at: http://lxr.free-electrons.com/source/fs/btrfs/volumes.c#L848 causing a dmesg like:

...
[62030.117248] btrfs: device label label devid 1 transid 13 /dev/dm-6
[62030.141242] btrfs: device label label devid 1 transid 13 /dev/dm-5
...

It is unclear exactly what is causing the "remount" or why it is needed. But the remarks that the duplicate UUID is responsible seem not far fetched.

I'm not even sure that this kind of remount (changing the device of existing mount-point) is wanted or useful behavior...

If you would want to change the behavior you could modify or remove the btrfs-udev rules with loss of functionality: no more auto-mounts after hot-plugging btrfs usb disks.

Answer 2

I have not looked massively into this but btrfs as a filesystem works on groups of disks, not individual devices.

I suspect that there is no way for btrfs to distinguish between the mounted snapshot and the real mounted filesystem when a mount occurs. It may as a matter of fact see the UUID of the underlying subvolume, assume its a mirror of the original volume and write to both volumes at the same time.

I would be surprised if this ever gets fixed seeing as for most intents and purposes btrfs snapshots supersede LVM snapshots.