How to run a command in another pod in another namespace from an Airflow worker in a Kubernetes cluster

Question

I want to run a papermill command in a pod and get the errors (if any). This has to be done from an Airflow worker using the bash operator i Airflow.

I have tried running the command as needed, I have tried running multiple bash commands from my own worker, commands that were meant to be executed inside the worker. But this just gives me the idea that something like this is possible. Although, if there is any better approach that would solve this, I am all ears.

The bash command to be run is:

papermill check1.ipynb check1_output.ipynb -p params '{"aviral":"srivastava", "apoorv":"srivastava"}'

Manually, I exec in my pod and then run this command. I want to make this a bash command in airflow's dag.

Has your Airflow DAG container Kubectl installed and running in same cluster? — Vishal Biyani, May 27 '19 at 12:46
@VishalBiyani No. How can I install Kubectl? Or should I install it? Or should I go with any python cli for the same? — Aviral Srivastava, May 27 '19 at 13:24
@VishalBiyani yes, the worker is running in the same cluster. — Aviral Srivastava, May 27 '19 at 13:27

score 0 · Answer 1 · answered May 27 '19 at 14:28

To use kubectl from within a pod you need to have binaries available in your pod.

You can either Create a Docker image with installation of kubectl.

I think the Dockerfile might look like this:

FROM ubuntu:14.04

# Install.
RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update && \
  apt-get -y upgrade && \
  apt-get install -y build-essential && \
  apt-get install -y software-properties-common && \
  apt-get install -y byobu curl git htop man unzip vim wget && \
  rm -rf /var/lib/apt/lists/* && \
# Installing kubectl using native package management  
  apt-get install -y apt-transport-https && \
  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee -a /etc/apt/sources.list.d/kubernetes.list && \
  apt-get update && \
  apt-get install -y kubectl

# Add files.
ADD root/.bashrc /root/.bashrc
ADD root/.gitconfig /root/.gitconfig
ADD root/.scripts /root/.scripts

# Set environment variables.
ENV HOME /root

# Define working directory.
WORKDIR /root

# Define default command.
CMD ["bash"]

Or Define a Command and Arguments for a Container and use it to install kubectl when pod is starting.

If you want to use it from within Kubernetes Deployment the part of .yaml might look like this:

...
spec:
  containers:
  - name: ubuntu-with-kubectl
    image: ubuntu
    command: ["/bin/sh","-c"]
    args:
      - apt-get update && apt-get install -y apt-transport-https;
        curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -;
        echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | tee -a /etc/apt/sources.list.d/kubernetes.list;
        apt-get update;
        apt-get install -y kubectl;
...

You can also kubectl exec -it <pod_name> bash into the pod and install is manually using this guide Install and Set Up kubectl, but it will be gone on container restart.

How to run a command in another pod in another namespace from an Airflow worker in a Kubernetes cluster

1 Answers1