How do I pass python scripts to a container while I am creating it in K8S?

Question

I have Apache Nifi working in Kubernetes. I have deployed it with yaml file. I have several Python scripts I would like to call on Apache Nifi.

I used this yaml file to deploy Apache Nifi:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nifi
  namespace: test-namespace
  labels:
    name : test-nifi
    app : test-nifi
spec:
  replicas: 2
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: test-nifi
  template:
    metadata:
      labels:
        app: test-nifi
    spec:
      restartPolicy: Always
      containers:
      - name: nifi1
        image: XXX
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8443
          name: nifi1
        env:
        - name: "NIFI_SENSITIVE_PROPS_KEY"
          value: "..."
        - name: ALLOW_ANONYMOUS_LOGIN
          value: "no"
        - name: SINGLE_USER_CREDENTIALS_USERNAME 
          value: XXX
        - name: SINGLE_USER_CREDENTIALS_PASSWORD 
          value: XXX
        - name: NIFI_WEB_HTTPS_HOST
          value: "0.0.0.0"
        - name: NIFI_WEB_HTTPS_PORT
          value: "8443"
        - name: NIFI_WEB_PROXY_HOST
          value: 0.0.0.0:8443
        - name: HOSTNAME
          value: "nifi1"
        - name: NIFI_ANALYTICS_PREDICT_ENABLED
          value: "true"
        - name: NIFI_ELECTION_MAX_CANDIDATES
          value: "1"
        - name: NIFI_ELECTION_MAX_WAIT
          value: "20 sec"
        - name: NIFI_JVM_HEAP_INIT
          value: "3g"
        - name: NIFI_JVM_HEAP_MAX
          value: "4g"
        volumeMounts:
          - name: pv-01
            mountPath: /opt/nifi/nifi-current/data
            subPath: data
        livenessProbe:
          exec:
            command:
              - pgrep
              - java
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 10
          failureThreshold: 3
          successThreshold: 1
        readinessProbe:
          tcpSocket:
              port: 8443
          initialDelaySeconds: 240
          periodSeconds: 30
          timeoutSeconds: 10
          failureThreshold: 3
          successThreshold: 1
        resources:
          requests:
            cpu: 400m
            memory: 4Gi
          limits:
            cpu: 500m
            memory: 4Gi
      imagePullSecrets:
      - name: repo-secret
      volumes:
        - name: pv-01
          persistentVolumeClaim:
            claimName: pv-01-claim

Solution I have :

Inject these scripts as config maps, this way the Apache Nifi container will have access to scripts.

How can I do it with config maps ?

Evgeny K · Answer 1 · 2022-11-23T15:28:16.420

1

To inject scripts as config map you should:

Create kubernetes ConfigMap and paste python script you want to have

apiVersion: v1
kind: ConfigMap
metadata:
  name: pyhton-scripts-cofigmap
data:
  python_script_1.py: |
    import foo from bar
    
    foo()

  python_script_2.py: |
    print('Hello World')

Attach this ConfigMap as volume to pod in your Deployment

in volumes: section of your deployment add

- configMap:
      name: pyhton-scripts-cofigmap
  name: pyhton-scripts-volume

it will be look

volumes:
  - name: pv-01
    persistentVolumeClaim:
       claimName: pv-01-claim
  - configMap:
       name: pyhton-scripts-cofigmap
    name: pyhton-scripts-volume

and in volumeMounts: section add

- mountPath: /path/to/scripts
  name: pyhton-scripts-volume
  readOnly: true

it will be look like

volumeMounts:
   - name: pv-01
     mountPath: /opt/nifi/nifi-current/data
     subPath: data
   - mountPath: /path/to/scripts
     name: pyhton-scripts-volume
     readOnly: true

You can create ConfigMap separately or add to your yaml like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: pyhton-scripts-cofigmap
data:

...

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nifi
  namespace: test-namespace
  labels:
    name : test-nifi
    app : test-nifi

...

---

edited Nov 23 '22 at 15:28

answered Nov 23 '22 at 15:21

Evgeny K

1,744
3
13
19

Thank you. Do you think the other idea of creating a python container inside the same pod of Apache Nifi container can be a good idea ? I don't how the Apache Nifi container can next call the other container having scripts. – lbened Nov 23 '22 at 15:28
oh, it's very much depends when it should call other container and how. Pod-to-pod communication is not simple in kubernetes. – Evgeny K Nov 23 '22 at 15:35
Thanks, so maybe config maps it's the best solution. But how can I do in config maps to give him several files that are in a specific tree structure? (I use Python classes, one main script will call others) – lbened Nov 23 '22 at 16:40
ConfigMap is not designed for that purpose. As workaround you can follow this [advice](https://stackoverflow.com/a/55790544). Another option will be to build image with those files (if it's possible) – Evgeny K Nov 23 '22 at 16:50
Yes I think it is possible. How the containers can communicate if they are in the same pod ? – lbened Nov 23 '22 at 16:58
See this [article](https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/) and [this](https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/) – Evgeny K Nov 23 '22 at 17:12

How do I pass python scripts to a container while I am creating it in K8S?

1 Answers1