0

this might sound like a very silly question for K8's experts. But I have been struggling with this for a while, thus the question below.

I'm trying to deploy locally a simple Kubernetes application through Minikube and docker to test the sidecar container pattern.

Let's start with the sidecar container elements:

Dockerfile

FROM python:3.6-alpine

RUN pip install boto3==1.14.20
RUN pip install --upgrade awscli s3cmd
   
ENV APP_HOME_DIR=/home/sidecar
RUN mkdir $APP_HOME_DIR

ADD run.sh $APP_HOME_DIR
ADD s3_sync.py $APP_HOME_DIR
RUN chmod +x $APP_HOME_DIR/s3_sync.py
RUN chmod +x $APP_HOME_DIR/run.sh

ENV BUCKET_NAME=
ENV PROJECT_NAME=
ENV FEATURE_BRANCH=
ENV LOCAL_DIR=

WORKDIR $APP_HOME_DIR


ENTRYPOINT ["./run.sh"]

run.sh

#!/bin/sh

while true
do
    echo "Triggering an S3-SYNC"
    python /home/sidecar/s3_sync.py -b $BUCKET_NAME -p $PROJECT_NAME -f $FEATURE_BRANCH -l $LOCAL_DIR
    sleep 10
done

And a simplified version of the python file s3_sync.py:

import logging
import argparse

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.FileHandler("debug.log"), logging.StreamHandler()],
)

logger = logging.getLogger(__name__)

MASTER = "master"

def main():

    bucket_name = args.bucket_name
    project_name = args.project_name
    branch = args.feature_branch
    local_dir = args.local_dir
    sync_type = "production" if branch == MASTER else "review"

    logger.info(f"S3-SYNC starting for {sync_type} environment")



if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-b", "--bucket_name", help="s3 bucket name", required=True)
    parser.add_argument("-p", "--project_name", help="project name to sync", default="")
    parser.add_argument("-f", "--feature_branch", help="git feature branch name", default="master")
    parser.add_argument("-l", "--local_dir", help="local dir where to sync the s3 objects", required=True)

    args = parser.parse_args()
    main()

If I build the above and run it through:

docker build -t sidecar_python .
docker run -e BUCKET_NAME=MYBUCKET -e PROJECT_NAME=MYPROJECT -e FEATURE_BRANCH=MYBRANCH -e LOCAL_DIR=/home sidecar_python

things run smoothly.

I then combine the sidecar in the Kubernetes deployment yaml file with another local Docker image:

k8_deployment.yml

apiVersion: v1
kind: Pod
metadata:
  name: mc1
spec:
  volumes:
    - name: dags
      emptyDir: {}
  containers:
    - name: readdag
      image: dag_reader
      imagePullPolicy: Never
      volumeMounts:
        - name: dags
          mountPath: /usr
    - name: sidecar
      image: sidecar_python:latest
      imagePullPolicy: Never
      env:
        - name: BUCKET_NAME
          value: "MYBUCKET"
        - name: PROJECT_NAME
          value: "MYPROJECT"
        - name: FEATURE_BRANCH
          value: "MYBRANCH"
        - name: LOCAL_DIR
          value: "/usr"
      volumeMounts:
        - name: dags
          mountPath: /usr

The dag_reader image is another local image, to keep things short I won't post its Dockerfile as it's very simple and outside the scope of my question.

Then I run the usual: minikube start, eval $(minikube docker-env), docker build.. of the 2 images and in the end kubectl apply -f k8_deployment.yml.

If I execute:

kubectl get po
NAME   READY   STATUS    RESTARTS   AGE
mc1    2/2     Running   0          2m7s

things seem to look good.

But then if I check the logs of sidecar container:

kubectl logs -f mc1 sidecar

Triggering an S3-SYNC
./run.sh: line 6: python: not found

So python is not found, I can't find the reason why the container running in the pod can't find python, but the container running standalone in docker can.

The same error is thrown for the other container running in the pod: also that one has a python alpine image.

Can anyone explain why? Am I doing something wrong here?

Thanks, Alessio

AlessioG
  • 576
  • 5
  • 13
  • 32
  • 1
    You're mounting the `emptyDir` volume `dags` over the container's entire `/usr` directory, including things like `/usr/bin/python3.6`. That's sort of equivalent to a `docker run -v` option which your simplified `docker run` command doesn't have. Do you mean this to be mounted somewhere else? – David Maze Jul 14 '20 at 14:32
  • @DavidMaze good catch! I didn't think about that. Moving the volume to another directory fixed the issue. Thanks a lot! – AlessioG Jul 14 '20 at 14:43

1 Answers1

3

The /usr directory contains a variety of system and application software. In particular, the Python binary is typically in /usr/bin/python3 on a Linux system (or container).

Your Kubernetes YAML mounts an emptyDir volume over /usr. That hides everything that was in that directory tree, including the Python binary and all of the Python system libraries. That leads to this error.

Mounting the volume somewhere else would avoid this problem. Containerized applications tend to not be overly picky about "standard" FHS paths, so I might set instead

  env:
    - name: LOCAL_DIR
      value: "/dags"
  volumeMounts:
    - name: dags
      mountPath: /dags
David Maze
  • 130,717
  • 29
  • 175
  • 215