How to manage pod replicas while using PVC

Question

I have an Open Source project that I deployed in Kubernetes. It has a master, data, and query pod. The data pod is used to store the data. Unfortunately, when the data pod goes down it loses all its stored data as well. So I figured, I have to use a StatefulSet with PVC. I started looking at PVC samples. Suppose I create a PV:

kind: PersistentVolume
metadata:
  name: task-pv-volume1
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data1"

And then I have a Claim and a POD to bind it. This is the sample Stateful Set:

kind: StatefulSet
metadata:
  name: myweb
spec:
  replicas: 2
...
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local
      resources:
        requests:
          storage: 100Mi

Now when I run a sample job, I realize that only one of the 2 pods is active. The other pod fails saying:

Warning  FailedScheduling  <unknown>        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.

which I believe is because I created only one PV. So I go and create a new PV,

path: "/mnt/data2"

and the pod is up and running.

In my prod setup, I want this to happen automatically. Which means I do not want to create these PVs for each replica. Is there a way to do this? Is this taken care of by Kube in some way?

Appreciate the help. If you have any further questions, please reply. I'll try my best to resolve it.

Hi there. For a StatefulSet you could use a `volumeClaimTemplates` (https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/). This will result in each replica being given its own `PersistentVolumeClaim`. Each pod will get it's own disk. An empty disk. They won't automatically sync up or anything. What is your goal with the storage? Would you like multiple pods to have a synced up view of the disk? — Justin Tamblyn, Nov 12 '20 at 06:28
Thanks for your answer. I have used volumeClaimTemplates. And slightly modified the question to make it a bit simpler. My goal is I do not want to sync up. I want each pod to write to unique locations. But how do I do it automatically without manually created a PV for each replica? — CSUNNY, Nov 12 '20 at 16:07
Ah I see. Apologies for that. The PersistentVolumeClaim references a `storageClassName`. This is a reference to a `StorageClass` (another Kubernetes object). This is to allow a particular storage provider to setup the PersistentVolumes for you. So what you're looking for is a thing. It's handled by the provisioner of your storage class. Have you chosen a storage provisioner? If you're not sure, checkout https://github.com/rancher/local-path-provisioner Also just a heads up: You mention production. `local` as a storage class in production might have some shortcomings. — Justin Tamblyn, Nov 13 '20 at 06:16

score 1 · Answer 1 · answered Nov 13 '20 at 06:21

Currently the Kubernetes local storage provisioner is static and not dynamic. A dynamic provisioner would allow automatic spinning up of Volumes as mentioned in the post.

As the default Kubernetes local storage provisioner does not support this you will unfortuantely have to find another storage provisioner.

I would suggest looking at Ranchers local-path-provisioner: https://github.com/rancher/local-path-provisioner

It advertises in its overview

Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the user configuration, the Local Path Provisioner will create hostPath based persistent volume on the node automatically. It utilizes the features introduced by Kubernetes Local Persistent Volume feature, but make it a simpler solution than the built-in local volume feature in Kubernetes.

Compare to built-in Local Persistent Volume feature in Kubernetes

Pros

Dynamic provisioning the volume using hostPath. Currently the Kubernetes Local Volume provisioner cannot do dynamic provisioning for the local volumes.

Cons

No support for the volume capacity limit currently. The capacity limit will be ignored for now.

You can install it by running

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

and then changing your storageClassName in your stateful set to local-path

How to manage pod replicas while using PVC

1 Answers1