3

Im trying to apply a config file to create a POD from Cloud Compose using the BashOperator

First I tried using the PodOperator but it doesnt allow to pass a spec file, it just builds from the image.

I tried using the BashOperator since the worker already includes gcloud, kubectl, etc. https://cloud.google.com/composer/docs/how-to/using/writing-dags#bashoperator

But every time I run a command I get some error with the arguments or with gcloud sdk.

I tried running single kubectl commands and kubectl is indeed installed on the worker.

I tried setting the config for the cluster before but I got errors on that call too.

job = BashOperator(
    task_id='start',
    bash_command='kubectl apply -f
/home/airflow/gcs/dags/.../spec.yaml',
    env=prodigy_env
    )

I expect the pod to be created effectively with the config file but the actual output is

{bash_operator.py:124} INFO - unable to recognize - Path to file

*If you run a cat /path the file indeed exists

Benjamin Gil
  • 41
  • 3
  • I am running airflow locally and running kubectl commands will run on my local machine. Can you help how can I run kubectl commands on GCP using bash operator? – Vinit Bodhwani Sep 15 '21 at 01:27

3 Answers3

0

I forgot to answer, apparently passing env vars through the bash operator was messing up with the execution.

The path exists and I was able to run the script without any problems I just couldn't pass env variables that way.

I ended up passing them in the spec.yaml

Benjamin Gil
  • 41
  • 3
0

You are having issues probably because although you have gcloud auth configured you still have to get kubectl credentials for the cluster:

gcloud container clusters get-credentials $CLUSTER_NAME --zone $GCP_ZONE --project $PROJECT_ID

Run this command before kubectl apply -f /home/airflow/gcs/dags/.../spec.yaml

Alan Borsato
  • 248
  • 2
  • 13
-1

The content of DAGs bucket Buckets/<your bucket>/dags/ is copied to '/home/airflow/gcsfuse/dags/' on workers preserving the folder structure.

If your spec.yaml is stored in your DAGs bucket like this Buckets/<your bucket>/dags/<subfolder1>/spec.yaml then the file should be available under the following path /home/airflow/gcsfuse/dags/<subfolder1>/spec.yaml.

Just taking a look on the "/home/airflow/gcs/dags/.../spec.yaml'" path from the file system perspective - instead of "..." you probably should use ".." to refer to folder above "dags" folder.

--

On the other hand, there is an alternative to creating a Pod using explicitly invoking kubectl command - namely you could use KubernetesPodOperator - this web page describes how to use KubernetesPodOperator to create Pods

rafalbiegacz
  • 96
  • 5