Why I cannot read files from a shared PersistentVolumeClaim between containers in Kubernetes?

Question

I have a docker image felipeogutierrez/tpch-dbgen that I build using docker-compose and I push it to docker-hub registry using travis-CI.

version: "3.7"
services:
  other-images: ....
  tpch-dbgen:
    build: ../docker/tpch-dbgen
    image: felipeogutierrez/tpch-dbgen
    volumes:
      - tpch-dbgen-data:/opt/tpch-dbgen/data/
      - datarate:/tmp/
    stdin_open: true

and this is the Dockerfile to build this image:

FROM gcc AS builder
RUN mkdir -p /opt
COPY ./generate-tpch-dbgen.sh /opt/generate-tpch-dbgen.sh
WORKDIR /opt
RUN chmod +x generate-tpch-dbgen.sh && ./generate-tpch-dbgen.sh

In the end, this scripts creates a directory /opt/tpch-dbgen/data/ with some files that I would like to read from another docker image that I am running on Kubernetes. Then I have a Flink image that I create to run into Kubernetes. This image starts 3 Flink Task Managers and one stream application that reads files from the image tpch-dbgen-data. I think that the right approach is to create a PersistentVolumeClaim so I can share the directory /opt/tpch-dbgen/data/ from image felipeogutierrez/tpch-dbgen to my flink image in Kubernetes. So, first I have this file to create the PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tpch-dbgen-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi

Then, I am creating an initContainers to launch the image felipeogutierrez/tpch-dbgen and after that launch my image felipeogutierrez/explore-flink:1.11.1-scala_2.12:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-taskmanager
spec:
  replicas: 3
  selector:
    matchLabels:
      app: flink
      component: taskmanager
  template:
    metadata:
      labels:
        app: flink
        component: taskmanager
    spec:
      initContainers:
      - name: tpch-dbgen
        image: felipeogutierrez/tpch-dbgen
        #imagePullPolicy: Always
        env:
        command: ["ls"]
        # command: ['sh', '-c', 'for i in 1 2 3; do echo "job-1 `date`" && sleep 5s; done;', 'ls']
        volumeMounts:
        - name: tpch-dbgen-data
          mountPath: /opt/tpch-dbgen/data
      containers:
      - name: taskmanager
        image: felipeogutierrez/explore-flink:1.11.1-scala_2.12
        #imagePullPolicy: Always
        env:
        args: ["taskmanager"]
        ports:
        - containerPort: 6122
          name: rpc
        - containerPort: 6125
          name: query-state
        livenessProbe:
          tcpSocket:
            port: 6122
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: flink-config-volume
          mountPath: /opt/flink/conf/
        - name: tpch-dbgen-data
          mountPath: /opt/tpch-dbgen/data
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
      - name: flink-config-volume
        configMap:
          name: flink-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j-console.properties
            path: log4j-console.properties
      - name: tpch-dbgen-data
        persistentVolumeClaim:
          claimName: tpch-dbgen-data-pvc

The Flink stream application is starting but it cannot read the files on the directory /opt/tpch-dbgen/data of the image felipeogutierrez/tpch-dbgen. I am getting the error: java.io.FileNotFoundException: /opt/tpch-dbgen/data/orders.tbl (No such file or directory). It is strange because when I try to go into the container felipeogutierrez/tpch-dbgen I can list the files. So I suppose there is something wrong on my Kubernetes configuration. Does anyone know to point what I am missing on the Kubernetes configuration files?

$ docker run -i -t felipeogutierrez/tpch-dbgen /bin/bash
root@10c0944a95f8:/opt# pwd
/opt
root@10c0944a95f8:/opt# ls tpch-dbgen/data/
customer.tbl  dbgen  dists.dss  lineitem.tbl  nation.tbl  orders.tbl  part.tbl  partsupp.tbl  region.tbl  supplier.tbl

Also, when I list the logs of the container tpch-dbgen I can see the directory tpch-dbgen that I want to read. Although I cannot execute the command command: ["ls tpch-dbgen"] inside my Kubernetes config file.

$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
flink-jobmanager-n9nws               1/1     Running   2          17m
flink-taskmanager-777cb5bf77-ncdl4   1/1     Running   0          4m54s
flink-taskmanager-777cb5bf77-npmrx   1/1     Running   0          4m54s
flink-taskmanager-777cb5bf77-zc2nw   1/1     Running   0          4m54s
$ kubectl logs flink-taskmanager-777cb5bf77-ncdl4 tpch-dbgen
generate-tpch-dbgen.sh
tpch-dbgen

score 1 · Answer 1 · answered Sep 22 '20 at 11:28

Docker has an unusual feature where, under some specific circumstances, it will populate a newly created volume from the image. You should not rely on this functionality, since it completely ignores updates in the underlying images and it doesn't work on Kubernetes.

In your Kubernetes setup, you create a new empty PersistentVolumeClaim, and then mount this over your actual data in both the init and main containers. As with all Unix mounts, this hides the data that was previously in that directory. Nothing causes data to get copied into that volume. This works the same way as every other kind of mount, except the Docker named-volume mount: you'll see the same behavior if you change your Compose setup to do a host bind mount, or if you play around with your local development system using a USB drive as a "volume".

You need to make your init container (or something else) explicitly copy data into the directory. For example:

initContainers:
- name: tpch-dbgen
  image: felipeogutierrez/tpch-dbgen
  command:
    - /bin/cp
    - -a
    - /opt/tpch-dbgen/data
    - /data
  volumeMounts:
    - name: tpch-dbgen-data
      mountPath: /data # NOT the same path as in the image

If the main process modifies these files in place, you can make the command be more intelligent, or write a script into your image that only copies the individual files in if they don't exist yet.

It could potentially make more sense to have your image generate the data files at startup time, rather than at image-build time. That could look like:

FROM gcc
COPY ./generate-tpch-dbgen.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/generate-tpch-dbgen.sh
CMD ["generate-tpch-dbgen.sh"]

Then in your init container, you can run the default command (the generate script) with the working directory set to the volume directory

initContainers:
- name: tpch-dbgen
  image: felipeogutierrez/tpch-dbgen
  volumeMounts:
    - name: tpch-dbgen-data
      mountPath: /opt/tpch-dbgen/data # or anywhere really
  workingDir: /opt/tpch-dbgen/data    # matching mountPath

so, do you mean that I still need the `PersistentVolumeClaim` but it is better to generate the files of the `felipeogutierrez/tpch-dbgen` when it is starting, otherwise I will overwrite the directory `/opt/tpch-dbgen/data`? — Felipe, Sep 22 '20 at 12:08
Nothing in Kubernetes will automatically copy data into that volume. If you mount the newly created volume over the image contents, there's no way to get what was originally in the image. — David Maze, Sep 22 '20 at 14:03
I don't get it. Do I need the `PersistentVolumeClaim`? I created the container `tpch-dbgen` as you described, pointing to another directory `/data` and making my java file read from that directory, but I still get the same error. — Felipe, Sep 22 '20 at 15:03
If you're content to regenerate that data on every pod startup, you can use an `emptyDir` volume instead (without a PVC). Note that the YAML fragments I have here _only_ deal with the init container; the volume mount into the main container should work unchanged. — David Maze, Sep 22 '20 at 15:41
When I start the pod with `PersistentVolumeClaim` and list the files on the initContainer I can see all the files that the image generates for me. However, when I use the mount command, it overwrites that directory, as you said. Even if I am copying to a new directory `/data`, when I mount this directory `/data` Kubernetes overwrites. So, I don't get ho to use the `mountPath` right.... — Felipe, Sep 22 '20 at 15:51
your command `- /bin/cp - -a - /opt/tpch-dbgen/data - /data` does not copy any files to the `data` directory. — Felipe, Sep 22 '20 at 16:08
In your pod you define init container using your docker image, mount your volume in it in some path (ie. /target) but instead of running regular content of your container, run something like cp -r /your/dir/* /target/ which will initiate your directory with expected content and exit allowing further startup of the pod — Malgorzata, Sep 23 '20 at 10:19

score 1 · Accepted Answer · answered Sep 23 '20 at 09:55

I got to run the PersistentVolumeClaim and share it between pods. Basically I had to use a subPath property which I learned from this answer https://stackoverflow.com/a/43404857/2096986 and I am using a simple Job that I learned from this answer https://stackoverflow.com/a/64023672/2096986. The final results is below:

The Dockerfile:

FROM gcc AS builder

RUN mkdir -p /opt
COPY ./generate-tpch-dbgen.sh /opt/generate-tpch-dbgen.sh
WORKDIR /opt
RUN chmod +x /opt/generate-tpch-dbgen.sh
ENTRYPOINT ["/bin/sh","/opt/generate-tpch-dbgen.sh"]

and the script generate-tpch-dbgen.sh has to have this line in the end sleep infinity & wait to not finalize. The PersistentVolumeClaim is the same of the question. Then I create a Job with the subPath property.

apiVersion: batch/v1
kind: Job
metadata:
  name: tpch-dbgen-job
spec:
  template:
    metadata:
      labels:
        app: flink
        component: tpch-dbgen
    spec:
      restartPolicy: OnFailure
      volumes:
        - name: tpch-dbgen-data
          persistentVolumeClaim:
            claimName: tpch-dbgen-data-pvc
      containers:
        - name: tpch-dbgen
          image: felipeogutierrez/tpch-dbgen
          imagePullPolicy: Always
          volumeMounts:
          - mountPath: /opt/tpch-dbgen/data
            name: tpch-dbgen-data
            subPath: data

and I use it on the other deployment also with the subPath property.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-taskmanager
spec:
  replicas: 3
  selector:
    matchLabels:
      app: flink
      component: taskmanager
  template:
    metadata:
      labels:
        app: flink
        component: taskmanager
    spec:
      volumes:
      - name: flink-config-volume
        configMap:
          name: flink-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j-console.properties
            path: log4j-console.properties
      - name: tpch-dbgen-data
        persistentVolumeClaim:
          claimName: tpch-dbgen-data-pvc
      containers:
      - name: taskmanager
        image: felipeogutierrez/explore-flink:1.11.1-scala_2.12
        imagePullPolicy: Always
        env:
        args: ["taskmanager"]
        ports:
        - containerPort: 6122
          name: rpc
        - containerPort: 6125
          name: query-state
        livenessProbe:
          tcpSocket:
            port: 6122
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: flink-config-volume
          mountPath: /opt/flink/conf/
        - name: tpch-dbgen-data
          mountPath: /opt/tpch-dbgen/data
          subPath: data
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary

Kamol Hasan · Answer 3 · 2020-09-28T04:23:58.740

Your generate-tpch-dbgen.sh script is executed while building the docker image resulting those files in /opt/tpch-dbgen/data directory. So, when you run the image, you can see those files.

But the problem with k8s pvc, when you mount the volume (initially empty) to your containers, it replaces the /opt/tpch-dbgen/data directory along with the files in it.

Solution:

Don't execute the generate-tpch-dbgen.sh while building the docker image, rather execute it in the runtime. Then, the files will be created in the shared pv from the init container.

Something like below:

FROM gcc AS builder
RUN mkdir -p /opt
COPY ./generate-tpch-dbgen.sh /opt/generate-tpch-dbgen.sh
RUN chmod +x /opt/generate-tpch-dbgen.sh
ENTRYPOINT ["/bin/sh","/opt/generate-tpch-dbgen.sh"]

score 0 · Answer 4 · answered Sep 23 '20 at 13:28

0

Maybe the issue is the accessMode you set on your PVC. ReadWriteOnce means it can only be mounted by one POD.

See here for Details.

You could try to use ReadWriteMany.

answered Sep 23 '20 at 13:28

Kimses

707
7
17

Why I cannot read files from a shared PersistentVolumeClaim between containers in Kubernetes?

4 Answers4