1

TL;DR;

I'm seeing some discrepancies between what I setup using zfs, what lxd may be doing during the lxd init process, and what I expected to see.

I've been using LXD for about a month and have gone over the documentation and made several searches, but they don't appear to touch this topic.

Please look at the section below for more details about my questions:

  1. Can someone explain why I'm observing the mountpoint discrepancy between LXD and ZFS and what implications this may have, if any?
  2. Is LXD actually moving the tank/lxd dataset contents out of the ZFS Pool VDEVs from its mount location at /mnt/tank/lxd/... and into /var/lib/lxd/..., or is something else being done? (a) How can I confirm/verify either case? (b) Better yet, how can I get LXD to keep/put everything in the dataset I setup using zfs?
  3. I want everything that would normally be stored under /var/lib/lxd/ to be stored within the tank/lxd I specified during lxd init: How can I do that? (A good ZFS/LXD-way is preferable; I could mv /var/lib/lxd into the tank and ln -s the directory to point to /mnt/tank/..., but that seems like a bad hack and want to avoid it. There must be a better way.)

Thanks in advance.

Background

I'm using LXD 3.0.0 in Ubuntu 18.04, setting it up to use a pre-existing ZFS Pool and Dataset. This is how it looks before running lxd init:

$ sudo zpool create -m /mnt/tank tank raidz3 <6 storage SSDs omitted>
$ sudo zfs set compression=on tank
$ sudo zfs create tank/lxd
$ sudo zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
tank       216K  1.31T  43.7K  /mnt/tank
tank/lxd  41.7K  1.31T  41.7K  /mnt/tank/lxd

This looks as I expected. I then run lxd init and follow up as follows:

$ lxd init
[...]
Create a new ZFS pool? (yes/no) [default=yes]: no
Name of the existing ZFS pool or dataset: tank/lxd
[...]

This is what zfs list shows after completing the above:

$ sudo zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank                  589K  1.31T  41.7K  /mnt/tank
tank/lxd              250K  1.31T  41.7K  none
tank/lxd/containers  41.7K  1.31T  41.7K  none
tank/lxd/custom      41.7K  1.31T  41.7K  none
tank/lxd/deleted     41.7K  1.31T  41.7K  none
tank/lxd/images      41.7K  1.31T  41.7K  none
tank/lxd/snapshots   41.7K  1.31T  41.7K  none

LXD created some datasets, but the mountpoints are no longer what they used to be (i.e. tank/lxd is gone) nor what I expected (nothing under lxd/... is listed). In short, the original mountpoint is gone and the other datasets are not showing up as I expected.

I'm probably wrong here, but it looks as if LXD has moved the data that should be under /mnt/tank/lxd/... out of the ZFS pool vdevs and into the /var/lib/lxd/ directory on the LVM where the OS was installed. I'm not sure what I've missed.

I can use zfs set commands to change "correct" this:

$ sudo zfs set mountpoint=/mnt/tank/lxd tank/lxd
$ sudo zfs set mountpoint=/mnt/tank/lxd/containers tank/lxd/containers
$ sudo zfs set mountpoint=/mnt/tank/lxd/custom tank/lxd/custom
$ sudo zfs set mountpoint=/mnt/tank/lxd/deleted tank/lxd/deleted
$ sudo zfs set mountpoint=/mnt/tank/lxd/images tank/lxd/images
$ sudo zfs set mountpoint=/mnt/tank/lxd/snapshots tank/lxd/snapshots

And verify that they show up as expected once again:

$ sudo zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank                  641K  1.31T  43.7K  /mnt/tank
tank/lxd              252K  1.31T  43.7K  /mnt/tank/lxd
tank/lxd/containers  41.7K  1.31T  41.7K  /mnt/tank/lxd/containers
tank/lxd/custom      41.7K  1.31T  41.7K  /mnt/tank/lxd/custom
tank/lxd/deleted     41.7K  1.31T  41.7K  /mnt/tank/lxd/deleted
tank/lxd/images      41.7K  1.31T  41.7K  /mnt/tank/lxd/images
tank/lxd/snapshots   41.7K  1.31T  41.7K  /mnt/tank/lxd/snapshots

But this seems like a partial solution, as all the other files (e.g. lxd.db, keys, etc) are left behind under /var/lib/lxd. So, I'm not sure if I'm "fixing" anything or if there's anything to fix at all.

In addition, I'm not clear on whether I actually achieved anything with the zfs set commands. For example, touch /var/lib/lxd/containers/hi.txt does not show the file under /mnt/tank/lxd/containers/, which I would've expected if they were both looking at the same location.

Thanks in advance for your help.

EDIT0: Suggested Test in Comments & Results

It was suggested that I try writing out some files as a test to see where the used space is actually consumed based on system info. The test steps were as follows:

  1. Destroy and re-create ZFS pool and LXD configs, but without running the zfs set mountpoint commands to let LXD do its own thing and see what happens. (Steps omitted below for brevity.)
  2. As a control, check pool/dataset storage capacity and usage on host.
  3. Enter a container and do another control check on space usage.
  4. Write out a test file inside container.
  5. Repeat checks on steps 1 & 2 to look for changes.

The results were inconclusive and can be seen below.

2. Control Check on the Host

$ sudo zfs list
NAME                                                                               USED  AVAIL  REFER  MOUNTPOINT
tank                                                                              1.18G  1.31T  41.7K  /mnt/tank
tank/lxd                                                                          1.18G  1.31T  41.7K  none
tank/lxd/containers                                                                270M  1.31T  41.7K  none
tank/lxd/containers/base-server                                                    251M  1.31T   588M  /var/lib/lxd/storage-pools/lxd-pool/containers/base-server
tank/lxd/containers/dev-server                                                    9.18M  1.31T   588M  /var/lib/lxd/storage-pools/lxd-pool/containers/dev-server
tank/lxd/containers/qa-server                                                     9.14M  1.31T   588M  /var/lib/lxd/storage-pools/lxd-pool/containers/qa-server
tank/lxd/custom                                                                    125K  1.31T  41.7K  none
tank/lxd/custom/dev-volume                                                        41.7K   512G  41.7K  /var/lib/lxd/storage-pools/lxd-pool/custom/dev-volume
tank/lxd/custom/qa-volume                                                         41.7K   512G  41.7K  /var/lib/lxd/storage-pools/lxd-pool/custom/qa-volume
tank/lxd/deleted                                                                  41.7K  1.31T  41.7K  none
tank/lxd/images                                                                    937M  1.31T  41.7K  none
tank/lxd/images/ab23ac2bda8cfb48c176a2d1e790181d5a1ed3522732a0cf8ff189dea848d3f1   588M  1.31T   588M  none
tank/lxd/images/b36ec647e374da4816104a98807633a2cc387488083d3776557081c4d0333618   349M  1.31T   349M  none
tank/lxd/snapshots                                                                41.7K  1.31T  41.7K  none

3. Inside a Container

root@dev-server:~$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
tank/lxd/containers/dev-server  1.4T  589M  1.4T   1% /
none                            492K     0  492K   0% /dev
udev                             32G     0   32G   0% /dev/tty
tmpfs                           100K     0  100K   0% /dev/lxd
tank/lxd/custom/dev-volume      512G  128K  512G   1% /mnt/tank
tmpfs                           100K     0  100K   0% /dev/.lxd-mounts
tmpfs                            32G     0   32G   0% /dev/shm
tmpfs                            32G  160K   32G   1% /run
tmpfs                           5.0M     0  5.0M   0% /run/lock
tmpfs                            32G     0   32G   0% /sys/fs/cgroup

4. Write out Test File in Container

root@dev-server:~$ dd if=/dev/zero of=/root/test.img bs=8MB
^C3929+0 records in
3929+0 records out
31432000000 bytes (31 GB, 29 GiB) copied, 17.7005 s, 1.8 GB/s

root@dev-server:~$ ls -lh
total 512
-rw-r--r-- 1 root root 30G May  7 20:21 test.img

5.1. Check Results from Container

root@dev-server:~$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
tank/lxd/containers/dev-server  1.4T  589M  1.4T   1% /
none                            492K     0  492K   0% /dev
udev                             32G     0   32G   0% /dev/tty
tmpfs                           100K     0  100K   0% /dev/lxd
tank/lxd/custom/dev-volume      512G  128K  512G   1% /mnt/tank
tmpfs                           100K     0  100K   0% /dev/.lxd-mounts
tmpfs                            32G     0   32G   0% /dev/shm
tmpfs                            32G  160K   32G   1% /run
tmpfs                           5.0M     0  5.0M   0% /run/lock
tmpfs                            32G     0   32G   0% /sys/fs/cgroup

5.2. Check Results from Host

$ sudo zfs list
NAME                                                                               USED  AVAIL  REFER  MOUNTPOINT
tank                                                                              1.18G  1.31T  41.7K  /mnt/tank
tank/lxd                                                                          1.18G  1.31T  41.7K  none
tank/lxd/containers                                                                270M  1.31T  41.7K  none
tank/lxd/containers/base-server                                                    251M  1.31T   588M  /var/lib/lxd/storage-pools/lxd-pool/containers/base-server
tank/lxd/containers/dev-server                                                    9.18M  1.31T   588M  /var/lib/lxd/storage-pools/lxd-pool/containers/dev-server
tank/lxd/containers/qa-server                                                     9.14M  1.31T   588M  /var/lib/lxd/storage-pools/lxd-pool/containers/qa-server
tank/lxd/custom                                                                    125K  1.31T  41.7K  none
tank/lxd/custom/dev-volume                                                        41.7K   512G  41.7K  /var/lib/lxd/storage-pools/lxd-pool/custom/dev-volume
tank/lxd/custom/qa-volume                                                         41.7K   512G  41.7K  /var/lib/lxd/storage-pools/lxd-pool/custom/qa-volume
tank/lxd/deleted                                                                  41.7K  1.31T  41.7K  none
tank/lxd/images                                                                    937M  1.31T  41.7K  none
tank/lxd/images/ab23ac2bda8cfb48c176a2d1e790181d5a1ed3522732a0cf8ff189dea848d3f1   588M  1.31T   588M  none
tank/lxd/images/b36ec647e374da4816104a98807633a2cc387488083d3776557081c4d0333618   349M  1.31T   349M  none
tank/lxd/snapshots                                                                41.7K  1.31T  41.7K  none

Running the df -h command on the host before and after the test file had been created/destroyed also showed no changes. I repeated this with a larger (100GB) test file, but failed to notice a difference.

code_dredd
  • 5,915
  • 1
  • 25
  • 53

1 Answers1

1

When mountpoint is none, it can mean one of two things:

  1. Someone explicitly unmounted that file system through the zfs unmount command.
  2. Someone used the mount command to set where that file system should be mounted, instead of using the zfs mount command.

I’d guess LXD is doing the latter, and ZFS just doesn’t know where it has been mounted. If so, LXD is probably also writing entries to /etc/fstab to recreate those mounts in /var/lib/lxd/... on reboot. (Heck, maybe the authors of LXD even consider it a feature that nosy ZFS enthusiasts will be less likely to try to store or modify things in their VM data directories if the mountpoint is less convenient to type in.)

So the short answer is that I think there was nothing to fix. However, I don’t think the change you made with zfs set will do any harm either.

You mentioned that touch ... in the LXD directories doesn’t create anything in the /mnt/tank/... directories. A file system can only be mounted in one place at a time, so I think the corresponding directories under /mnt are just directories that ZFS left in place when the mountpoint got changed. If you try to switch the mountpoint back to /mnt/... now, it should force you to delete the file you touched there before it allows the mount to happen (“overlaying” duplicate directories from two different file systems isn’t allowed). If you want them to be linked for some reason, you can use a symlink to do that.

In the future, you can figure out what file system a file is written on using df -P /path/to/file to help debug issues like this.

Dan
  • 7,155
  • 2
  • 29
  • 54
  • LXD does umount the file system, but does not _seem_ to be using a `zfs` cmd; otherwise, I'd expect `zfs mount` to be used later and `zfs list` to show the mountpoint, but it's not the case. Creating a volume w/ `lxc storage volume create ...` makes `zfs list` show the `mountpoint` as being under `/var/lib/lxd/storage-pools/...`. No entries are added to `/etc/fstab`. Checking w/ `df -P /var/lib/lxd/...` shows the files mounted on root `/` rather than `/mnt/...`, _suggesting_ the data is now _outside_ the ZFS pool, but it's only a mountpoint, so it's still unclear to me. Can you clarify that? – code_dredd May 02 '18 at 17:58
  • Right, my guess was that lxd is mounting the file system in some way that doesn’t work great with ZFS’s built-in accounting; I guess sometimes it could do the “right” / intuitive thing though, maybe because creating a file system will set the built-in accounting and also mount the file system. One last (weird) possibility I thought of is that lxd could be making an empty parent file system to set up some shared properties, and then leaving it unmounted since it will create child file systems for the volumes you create (and only mount those). – Dan May 04 '18 at 08:44
  • Due to the behavior, I'm currently taking the time to explicitly mount each dataset under the pool's mountpoint to make sure that ZFS is at least listing them correctly. That said, I haven't been able to actually confirm or disprove anything in a definitive manner. I'd like to avoid guessing/assuming that something is (likely) the case, only to later find out it was wrong and end up with a nasty surprise. Do you have any suggestions to test/confirm which one may actually be taking place? – code_dredd May 04 '18 at 20:20
  • I guess I would try writing some stuff in one of the volumes and seeing if zfs list shows any additional space usage. – Dan May 04 '18 at 20:24
  • I'll give this a try when I get back to the office next week and see how it goes. I'll share whatever I find so that the answer can account for it and any other attempts and/or findings to be more complete. – code_dredd May 05 '18 at 00:28
  • I tried some stuff out, but the results were odd. Please see my edit in the main post. – code_dredd May 07 '18 at 20:47
  • BTW, the test was performed after destroying and re-creating everything at both the ZFS Pool and LXD levels. I avoided the `zfs set mountpoint` commands to observe what LXD might've been doing. – code_dredd May 07 '18 at 20:50
  • 1
    Try copying from /dev/urandom instead of /dev/zero inside the containers -- the blocks of zeroes will be compressed to almost 0 use, so might not show up at the resolution you're looking at. – Dan May 07 '18 at 20:55
  • Ack, of course... indeed, sourcing it from `/dev/urandom` was the way to go for the test to show conclusive results. For some reason, the fact that I had enabled compression in the ZFS pool last week and that this would lead to all the `/dev/zero`es getting compressed into a negligible amount escaped me. The data _is_ being kept within the pool as expected, but now I was able to confirm it directly. You should consider updating your answer. Thanks a lot for the help. – code_dredd May 07 '18 at 21:04
  • _"So the short answer is that I think there was nothing to fix."_. Although the details were not completely correct, this sentence above turned out to be precisely right. Your comments were also pretty helpful and I appreciate it. I expect to add an answer with more detailed information for posterity. – code_dredd May 08 '18 at 16:07