5

Basically, I'm creating a StatefulSet deployment with 2 pods (single host cluster), I would like to that each pod will be able to mount to a base folder in the host, and to a subfolder beneath it:

Base folder mount: /mnt/disks/ssd

Pod#1 - /mnt/disks/ssd/pod-1

Pod#2 - /mnt/disks/ssd/pod-2

I've managed only to mount the first pod to the base folder, but the 2nd folder cannot mount (as the volume is already taken)

This is the volume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv
spec:
  capacity:
    storage: 5Gi 
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntukuber

This is the usage in the stateful set:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: app
  namespace: test-ns
spec:
  serviceName: app
  replicas: 2
....
....
        volumeMounts:
          - name: data
            mountPath: /var/lib/app/data
volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "local-storage"
        resources:
          requests:
            storage: 2Gi

So, i basically would like that each replica would use its own subfolder - how can one achieve it?

== EDIT ==

I've made some progress, i'm able to mount several replicas into the same mount, using the following YAMLs (the app i'm trying to do it on is rabbitmq - so i'll leave the app name as is)

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-local
  namespace: test-rabbitmq
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 6Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/disks"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: hostpath-pvc
  namespace: test-rabbitmq
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  selector:
    matchLabels:
      type: local
---

In the StatefulSet i'm declaring this volume:

  volumes:
    - name: rabbitmq-data
      persistentVolumeClaim:
        claimName: hostpath-pvc

And mounting "rabbitmq-data".

Both pods mount to the same folder, but will not create subfolders - this is no terrible situation as by default there are rabbitmq's subfolders - i'll try to expand it into each pod to use a subfolder

ArielB
  • 1,184
  • 2
  • 11
  • 36

1 Answers1

5

I am able to achieve the above scenario, what you need is "claimRef" in your pv to bind your PVC. Please have a look at following pv json and statefulset json

PV-0.json

{
  "kind": "PersistentVolume",
  "apiVersion": "v1",
  "metadata": {
    "name": "pv-data-vol-0",
    "labels": {
      "type": "local"
    }
  },
  "spec": {
    "capacity": {
      "storage": "10Gi"
    },
    "accessModes": [
      "ReadWriteOnce"
    ],
    "storageClassName": "local-storage",
    "local": {
      "path": "/prafull/data/pv-0"
    },
    "claimRef": {
      "namespace": "default",
      "name": "data-test-sf-0"
    },
    "nodeAffinity": {
      "required": {
        "nodeSelectorTerms": [
          {
            "matchExpressions": [
              {
                "key": "kubernetes.io/hostname",
                "operator": "In",
                "values": [
                  "ip-10-0-1-46.ec2.internal"
                ]
              }
            ]
          }
        ]
      }
    }
  }
}

PV-1.json

{
  "kind": "PersistentVolume",
  "apiVersion": "v1",
  "metadata": {
    "name": "pv-data-vol-1",
    "labels": {
      "type": "local"
    }
  },
  "spec": {
    "capacity": {
      "storage": "10Gi"
    },
    "accessModes": [
      "ReadWriteOnce"
    ],
    "storageClassName": "local-storage",
    "local": {
      "path": "/prafull/data/pv-1"
    },
    "claimRef": {
      "namespace": "default",
      "name": "data-test-sf-1"
    },
    "nodeAffinity": {
      "required": {
        "nodeSelectorTerms": [
          {
            "matchExpressions": [
              {
                "key": "kubernetes.io/hostname",
                "operator": "In",
                "values": [
                  "ip-10-0-1-46.ec2.internal"
                ]
              }
            ]
          }
        ]
      }
    }
  }
}

Statefulset.json

{
  "kind": "StatefulSet",
  "apiVersion": "apps/v1beta1",
  "metadata": {
    "name": "test-sf",
    "labels": {
      "state": "test-sf"
    }
  },
  "spec": {
    "replicas": 2,
    "template": {
      "metadata": {
        "labels": {
          "app": "test-sf"
        },
        "annotations": {
          "pod.alpha.kubernetes.io/initialized": "true"
        }
      }
      ...
      ...
    },
    "volumeClaimTemplates": [
      {
        "metadata": {
          "name": "data"
        },
        "spec": {
          "accessModes": [
            "ReadWriteOnce"
          ],
          "storageClassName": "local-storage",
          "resources": {
            "requests": {
              "storage": "10Gi"
            }
          }
        }
      }
    ]
  }
}

There will be two pods created test-sf-0 and test-sf-1 which in-turn will be created two PVC data-test-sf-0 and data-test-sf-1 which will be bound to PV-0 and Pv-1 respectively. Hence test-sf-0 will write to the location specified in PV-0 and test-sf-1 will write in location specified on PV-1. Hope this helps.

Prafull Ladha
  • 12,341
  • 2
  • 37
  • 58
  • Gosh! this was it :) Thank you! could you explain about the claim some more? does it concat the "data" volume name with the pod name (data-rabbitmq-0 for example)? – ArielB Oct 23 '18 at 15:46
  • You're welcome :) claimRef is used to bind your PV to a particular PVC. Whenever you define PVC as volumeClaimTemplates, the name of the PVC will be the name you provided and followed by dash and pod name e.g data-test-sf-0, data-test-sf-1. So by using claimRef in PV you can decide which PVC you want to bind the PV. – Prafull Ladha Oct 23 '18 at 16:27
  • Also, I have seen your edit regarding the subfolder, currently local-storage doesn't support automatically creation of sub-folders. So, whatever path you provide in PV must exist. ```"local": { "path": "/prafull/data/pv-0" },``` – Prafull Ladha Oct 23 '18 at 16:28
  • Even if i do create the subfolders on purpose? There’s currently no way then to bind a pvc to 2 replicas where the subfolders exist? Because ive tried using env variables but they were not interpreted at all. Any read on how the pv, pvc binding logic works would be also appriciated :) – ArielB Oct 23 '18 at 21:54
  • @PrafullLadha Specifying node names under `nodeAffinity` in persistent volume spec doesn't seem to guarantee scheduling of pods on the right node, despite what the official [docs](https://kubernetes.io/docs/concepts/storage/volumes/#local) say on local volumes. From time to time I face issues with `StatefulSet` pods claims binding to wrong persistent volumes. I wonder if this is an issue specific to using `volumeClaimTemplates` with StatefulSets like provided in your above `StatefulSet.json` example. – joe Aug 18 '19 at 04:35