Kubernetes copying jars into a pod and restart

Question

I have a Kubernetes problem where I need to copy 2 jars (each jar > 1Mb) into a pod after it is deployed. So ideally the solution is we cannot use configMap (> 1Mb) but we need to use "wget" in "initcontainer" and download the jars. so below is my kubernetes-template configuration which i have modified. The original one is available at https://github.com/dremio/dremio-cloud-tools/blob/master/charts/dremio/templates/dremio-executor.yaml

{{ if not .Values.DremioAdmin }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dremio-executor
spec:
  serviceName: "dremio-cluster-pod"
  replicas: {{.Values.executor.count}}
  podManagementPolicy: "Parallel"
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: dremio-executor
  template:
    metadata:
      labels:
        app: dremio-executor
        role: dremio-cluster-pod
      annotations:
        dremio-configmap/checksum: {{ (.Files.Glob "config/*").AsConfig | sha256sum }}
    spec:
      terminationGracePeriodSeconds: 5
      {{- if .Values.nodeSelector }}
      nodeSelector:
        {{- range $key, $value := .Values.nodeSelector }}
        {{ $key }}: {{ $value }}
        {{- end }}
      {{- end }}
      containers:
      - name: dremio-executor
        image: {{.Values.image}}:{{.Values.imageTag}}
        imagePullPolicy: IfNotPresent
        securityContext:
          runAsUser: 0
        resources:
          requests:
            memory: {{.Values.executor.memory}}M
            cpu: {{.Values.executor.cpu}}
        volumeMounts:
        - name: dremio-executor-volume
          mountPath: /opt/dremio/data
       ##################### START added this section #####################
        - name: dremio-connector
          mountPath: /opt/dremio/jars
       #################### END added this section ##########################
        - name: dremio-config
          mountPath: /opt/dremio/conf
        env:
        - name: DREMIO_MAX_HEAP_MEMORY_SIZE_MB
          value: "{{ template "HeapMemory" .Values.executor.memory }}"
        - name: DREMIO_MAX_DIRECT_MEMORY_SIZE_MB
          value: "{{ template "DirectMemory" .Values.executor.memory }}"
        - name: DREMIO_JAVA_EXTRA_OPTS
          value: >-
            -Dzookeeper=zk-hs:2181
            -Dservices.coordinator.enabled=false
            {{- if .Values.extraStartParams }}
            {{ .Values.extraStartParams }}
            {{- end }}
        command: ["/opt/dremio/bin/dremio"]
        args:
        - "start-fg"
        ports:
        - containerPort: 45678
          name: server
      initContainers:
      ################ START added this section ######################
      - name: installjars
        image: {{.Values.image}}:{{.Values.imageTag}}
        imagePullPolicy: IfNotPresent 
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: dremio-connector
          mountPath: /opt/dremio/jars
        command: ["/bin/sh","-c"]
        args: ["wget --no-check-certificate -O /dir/connector.jar https://<some nexus repo URL>/connector.jar; sleep 10;"]
      ################ END added this section ###############
      - name: wait-for-zk
        image: busybox
        command:  ["sh", "-c", "until ping -c 1 -W 1 zk-hs > /dev/null; do echo waiting for zookeeper host; sleep 2; done;"]
      # since we're mounting a separate volume, reset permission to
      # dremio uid/gid
      - name: chown-data-directory
        image: {{.Values.image}}:{{.Values.imageTag}}
        imagePullPolicy: IfNotPresent
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: dremio-executor-volume
          mountPath: /opt/dremio/data
        command: ["chown"]
        args:
        - "dremio:dremio"
        - "/opt/dremio/data"
      volumes:
      - name: dremio-config
        configMap:
          name: dremio-config
      {{- if .Values.imagePullSecrets }}
      imagePullSecrets:
        - name: {{ .Values.imagePullSecrets }}
      {{- end}}
     #################### START added this section ########################
      - name: dremio-connector
        emptyDir: {}
     #################### END added this section ########################
  volumeClaimTemplates:
  - metadata:
      name: dremio-executor-volume
    spec:
      accessModes: [ "ReadWriteOnce" ]
      {{- if .Values.storageClass }}
      storageClassName: {{ .Values.storageClass }}
      {{- end }}
      resources:
        requests:
          storage: {{.Values.executor.volumeSize}}
{{ end }}

So the above is NOT working and I don't see any jars being downloaded once I "exec" into the pod. I don't understand what is wrong with the above. however do note that inside the pod if i run the same wget command it does download the jar which baffles me. So the URL is working, readwrite of directory is no problem but still jar is not downloaded ???

I think in the long run is better to modify ( or build ) another image with jar included, so your kubernets config is more manageable — Vokail, Apr 30 '21 at 07:28
The `args:` looks wrong. You wrapped one large string into an array, but it should be an array of strings. Have you tried changing that? What is the error message you get? — Moritz Schmitz v. Hülst, May 04 '21 at 07:37

score 2 · Accepted Answer · answered Apr 30 '21 at 07:51

2

If you can remove the need for Wget altogether it would make life easier...

Option 1

Using your own docker image will save some pain if thats an option

Dockerfile

# docker build -f Dockerfile -t ghcr.io/yourOrg/projectId/dockerImageName:0.0.1 .
# docker push ghcr.io/yourOrg/projectId/dockerImageName:0.0.1

FROM nginx:1.19.10-alpine

# Use local copies of config
COPY files/some1.jar /dir/
COPY files/some2.jar /dir/

Files will be ready in the container, no need for cryptic commands in your pod definition that will make little sense. Alternatively if you need to download the files you could copy a script to do that work into the Docker image instead and run that on startup via the docker directive CMD.

Option 2

Alternatively, you could do a two stage deployment...

Create a persistent volume
mount the volume to a pod (use busybox as a base?) that will run for enough time for the files to copy across from your local machine (or for them to be downloaded if you continue to use Wget)
kubectl cp the files you need to the (Retained) PersistentVolume
Now mount the PV to your pod's container(s) so the files are readily available when the pod fires up.

answered Apr 30 '21 at 07:51

Rob Evans

2,822
1
9
15

I tried option 1 but it is failing i considered it though since the docker image is not build by me it asks for dependencies when doing the build so not much of an option. Option 2 can you please elaborate. Isnt it same as InitContainer option where we download the files and copy it into maincontainer before firing it up. Can you please elaborate on option 2. thanks. – Gururaj Nayak Apr 30 '21 at 13:05
instead of copying the files to a container, I'm suggesting you copy the files via a pod/container to a persistent volume mounted in the container.. then you can delete the pod, leave the PV in-tact as it can be "retained" and remounted on the container/pod you actually want to give access to the files. – Rob Evans Apr 30 '21 at 20:14
"@robevans" Good idea. will try it out. Thanks for the suggestion. – Gururaj Nayak May 01 '21 at 09:47
Thanks @robevans. Tried option 1 and somehow got it to work. Option 2 strangely did not work due to some PVC issue. – Gururaj Nayak May 04 '21 at 12:11
Perhaps you have the PVC access mode set to ReadWriteOnce rather than ReadWriteMany? – Rob Evans May 05 '21 at 09:33

score 1 · Answer 2 · answered Apr 30 '21 at 07:09

1

Your approch seems right. Another solution could be to include the jar on the Docker image but I think it's not possible right ?

You could just use an emptyDir instead of a VolumeClaim.

Last one, I would have download the jar before waiting for ZooKeeper to gain some time.

answered Apr 30 '21 at 07:09

XciD

2,537
14
40

I edited the question.my solution does not work even though wget is valid and once i do kubect exec into the pod and running the wget command manually does download the jar. It is just that initcontainer is having some issue downloading it. – Gururaj Nayak Apr 30 '21 at 07:30
Could you try with an emptyDir as mentioned ? – XciD Apr 30 '21 at 07:32
i added the emptyDir. It is throwing below error. Error: Could not find or load main class com.dremio.dac.daemon.DremioDaemon. Not sure what is wrong. Please inspect the file and let me know if you see something missing. If i comment out container:/volumeMounts:/ the dremio-connector section then it boots up but the file is not copied into the /opt/dremio/jars folder. If i change the directory to /opt/dremio/data everything works file is downloaded as well. Wierd issue. – Gururaj Nayak Apr 30 '21 at 14:30
To retrieve logs: `kubectl logs --follow dremio-executor-0 -p -c installjars` – XciD Apr 30 '21 at 14:47

Kubernetes copying jars into a pod and restart

2 Answers2

Option 1

Dockerfile

Option 2