How to make XFS quotas work in Kubernetes volumes on DigitalOcean?

Question

I need help with making the XFS quotas work in Kubernetes on DigitalOcean.

My problem essentially is that the xfs_quota tool seems to work only when one has also an access to the disk device, not only to the mounted volume. However, whatever I try, I can't seem to get access both to the device and the mount.

I tried both volume mounts and raw block volumes.

Volume Mounts

Here's my storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: block-storage-retain-xfs-prjquota
provisioner: dobs.csi.digitalocean.com
parameters:
  fsType: xfs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
mountOptions:
  - prjquota

Then I claim a new volume and add it to a pod like this:

volumeClaimTemplates:
  - metadata:
      name: my-storage
      namespace: my-namespace
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: block-storage-retain-xfs-prjquota
      resources:
        requests:
          storage: 1Gi

and mount it:

volumeMounts:
  - name: my-storage
    mountPath: "/var/www"

In the pod, everything gets mounted correctly, I have access to the volume (I can create contents in there) and the mount flag is set correctly:

$ mount | grep -i www
/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc on /var/www type xfs (rw,relatime,attr2,inode64,prjquota)

However, the disk device is not available in the pod:

$ ls -la /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc
ls: cannot access '/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc': No such file or directory

(in fact, the whole /dev/disk/ directory is not available)

According to my investigation, the lack of access to the device is what makes the XFS tools fail:

$ xfs_quota -x -c 'report -h' /var/www
xfs_quota: cannot setup path for mount /var/www: No such device or address

Raw Block Volumes

I also tried to switch to raw block volumes instead:

volumeClaimTemplates:
  - metadata:
      name: my-storage
      namespace: my-namespace
    spec:
      accessModes:
        - ReadWriteOnce
      volumeMode: Block
      storageClassName: block-storage-retain-xfs-prjquota
      resources:
        requests:
          storage: 1Gi

and add it as:

volumeDevices:
  - name: my-storage
    devicePath: /dev/my-storage

That gives me the device, but for some reason I can't format it / mount it (neither XFS nor ext4 actually):

$ mkfs.xfs /dev/my-storage
mkfs.xfs: error - cannot set blocksize 512 on block device /dev/my-storage: Permission denied

$ mkfs.ext4 /dev/my-storage
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 18f07181-737c-4b68-a5fe-ccd7f2c50ff8
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

$ mount /dev/my-storage /var/www
mount: /var/www: cannot mount /dev/my-storage read-only.

With SYS_ADMIN Linux capability, I can actually format it, but I'm still not able to mount it:

$ mkfs.xfs -f /dev/my-storage
meta-data=/dev/my-storage        isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

$ mount /dev/my-storage /var/www
mount: /var/www: cannot mount /dev/my-storage read-only.

(Why is the disk device read only?)

Raw Block Volume - With Partitions

Ok, so I tried to create a partition and format that. Partition is created successfully, but I don't have access to the partition devices:

$ fdisk -l /dev/my-storage
Disk /dev/my-storage: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk model: Volume
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb4a24142

Device            Boot Start     End Sectors  Size Id Type
/dev/my-storage1        2048 2097151 2095104 1023M 83 Linux

However, the /dev/my-storage1 does not exist:

$ ls -la /dev/my-storage*
brw-rw---- 1 root disk 8, 48 Oct 25 14:42 /dev/my-storage

I tried running the container as privileged which gave me access to more devices in /dev, but then I didn't see my raw block volume device at all.

What Next?

As I see that, any of those would work for me:

1. Getting access to the underlying block device for volume mounts.
2. Access to the partition device so that I can mount it.
3. Ability to mount the raw block volume (e.g. by making it not read-only, whatever it means?).
4. Making the xfs_quota tool NOT require the underlying device.

I believe I made it work a few months ago using raw block volumes with partitions, but either I forgot how or something changed on DigitalOcean and I can't seem to be able to create and access partitions anymore.

Any help is hugely appreciated, thank you!

Answer 1

Timo here from the Managed Kubernetes (DOKS) team at DigitalOcean.

What you are missing is the host system mount of the /dev directory. If you add both

 volumes:
 - name: device-dir
   hostPath:
     path: /dev

and

volumeMounts:
  - name: device-dir
    mountPath: /dev

to the manifest at the right places, things should work as expected.