Mounting NFS volume in Google Container Engine with Container OS (COS)

Question

After migrating the image type from container-vm to cos for the nodes of a GKE cluster, it seems no longer possible to mount a NFS volume for a pod.

The problem seems to be missing NFS client libraries, as a mount command from command line fails on all COS versions I tried (cos-stable-58-9334-62-0, cos-beta-59-9460-20-0, cos-dev-60-9540-0-0).

sudo mount -t nfs mynfsserver:/myshare /mnt

fails with

mount: wrong fs type, bad option, bad superblock on mynfsserver:/myshare,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

But this contradicts the supported volume types listed here: https://cloud.google.com/container-engine/docs/node-image-migration#storage_driver_support

Mounting a NFS volume in a pod works in a pool with image-type container-vm but not with cos.

With cos I get following messages with kubectl describe pod:

MountVolume.SetUp failed for volume "kubernetes.io/nfs/b6e6cf44-41e7-11e7-8b00-42010a840079-nfs-mandant1" (spec.Name: "nfs-mandant1") pod "b6e6cf44-41e7-11e7-8b00-42010a840079" (UID: "b6e6cf44-41e7-11e7-8b00-42010a840079") with: mount failed: exit status 1
Mounting command: /home/kubernetes/containerized_mounter/mounter
Mounting arguments: singlefs-1-vm:/data/mandant1 /var/lib/kubelet/pods/b6e6cf44-41e7-11e7-8b00-42010a840079/volumes/kubernetes.io~nfs/nfs-mandant1 nfs []
Output: Mount failed: Mount failed: exit status 32
Mounting command: chroot
Mounting arguments: [/home/kubernetes/containerized_mounter/rootfs mount -t nfs singlefs-1-vm:/data/mandant1 /var/lib/kubelet/pods/b6e6cf44-41e7-11e7-8b00-42010a840079/volumes/kubernetes.io~nfs/nfs-mandant1]
Output: mount.nfs: Failed to resolve server singlefs-1-vm: Temporary failure in name resolution

There seems to be a DNS problem on 'cos'. If I use the IP address of the NFS server on defining the persistent volume, it works on cos too — Martin Weindel, May 26 '17 at 08:14
I have the same issue, defining volumes using a direct ip works but using DNS names doesn't (tried hostnames, internal dns names as well as public dns). — Glenn Vandamme, Jul 04 '17 at 08:32

score 1 · Answer 1 · answered May 25 '17 at 22:30

1

Martin, are you setting up the mounts manually (executing mount yourself), or are you letting kubernetes do it on your behalf via a pod referencing an NFS volume?

The former will not work. The later will. As you've discovered COS does not ship with NFS client libraries, so GKE gets around this by setting up a chroot (at /home/kubernetes/containerized_mounter/rootfs) with the required binaries and calling mount inside that.

answered May 25 '17 at 22:30

Saad Ali

1,730
11
12

Saad, thanks for explaining the mechanics of mounting NFS volume of a pod. – Martin Weindel May 26 '17 at 07:52

score 1 · Answer 2 · answered Jul 05 '17 at 13:01

I've nicked the solution @saad-ali mentioned above, from the kubernetes project, to make this work.

To be concrete, I've added the following to my cloud-config:

# This script creates a chroot environment containing the tools needed to mount an nfs drive
- path: /tmp/mount_config.sh
  permissions: 0755
  owner: root
  content: |
    #!/bin/sh
    set +x # For debugging

    export USER=root
    export HOME=/home/dockerrunner
    mkdir -p /tmp/mount_chroot
    chmod a+x /tmp/mount_chroot
    cd /tmp/
    echo "Sleeping for 30 seconds because toolbox pull fails otherwise"
    sleep 30
    toolbox --bind /tmp /google-cloud-sdk/bin/gsutil cp gs://<uploaded-file-bucket>/mounter.tar /tmp/mounter.tar
    tar xf /tmp/mounter.tar -C /tmp/mount_chroot/
    mount --bind /tmp/mount_chroot /tmp/mount_chroot
    mount -o remount, exec /tmp/mount_chroot
    mount --rbind /proc /tmp/mount_chroot/proc
    mount --rbind /dev /tmp/mount_chroot/dev
    mount --rbind /tmp /tmp/mount_chroot/tmp
    mount --rbind /mnt /tmp/mount_chroot/mnt

The uploaded-file-bucket container the chroom image the kube team has created, downloaded from: https://storage.googleapis.com/kubernetes-release/gci-mounter/mounter.tar

Then, the runcmd for the cloud-config looks something like:

runcmd:
- /tmp/mount_config.sh
- mkdir -p /mnt/disks/nfs_mount
- chroot /tmp/mount_chroot /bin/mount -t nfs -o rw nfsserver:/sftp /mnt/disks/nfs_mount

This works. Ugly as hell, but it'll have to do for now.

Mounting NFS volume in Google Container Engine with Container OS (COS)

2 Answers2