Stack Overflow
Stack Overflow

Questions tagged [kubernetespodoperator]

38 questions
7
votes
4 answers

How to install dependency modules for airflow DAG task(or python code)? , "Failed to import module" in airflow DAG when using kuberentesExecutor

I have a airflow DAG "example_ml.py" which has a task "train_ml_model" and this task is calling/running a python script "training.py". -Dags/example_ml.py -Dags/training.py When I run Dag, it is failing to import modules required for training script…
veeresh patil
  • 1,168
  • 1
  • 11
  • 18
6
votes
0 answers

Airflow KubernetesPodOperator - using a ConfigMap value as an environmental variable

I have a Kubernetes ConfigMap called test that contains a key foobar with some value. I would like to use that key's value in an environmental variable. import datetime import os from airflow import models from airflow.contrib.kubernetes.volume…
P-S
  • 3,876
  • 1
  • 29
  • 26
4
votes
0 answers

Airflow KubernetesPodOperator : how to access secret passed to Pod?

I am trying to pass secret variables to my KubernetesPodOperator in airflow Here is what I have done : Create a secret.yaml file that looks like the following apiVersion: v1 kind: Secret metadata: name: my-secret type: Opaque data: SECRET_1:…
Luis Blanche
  • 557
  • 9
  • 18
3
votes
1 answer

Airflow - Unable to use jinja template for resources in Kubernetes Pod Operator task

Airflow - Unable to use jinja template for resources in Kubernetes Pod Operator task. Able to use jinja template for environment variables, image but not able to use for resources to specify CPU and memory (request and limit).
Raja Sekhar
  • 31
  • 1
2
votes
0 answers

Airflow 2.1.4 Composer V2 GKE kubernetes in custom VPC Subnet returning 404

So I have two V2 Composers running in the same project, the only difference in these two is that in one of them I'm using the default subnet and default values/autogenerated values for cluster-ipv4-cidr & services-ipv4-cidr. In the other one I've…
Anton
  • 581
  • 1
  • 5
  • 23
2
votes
1 answer

How to add custom component to Elyra's list of available airflow operators?

Trying to make my own component based on KubernetesPodOperator. I am able to define and add the component to the list of components but when trying to run it, I get: Operator 'KubernetesPodOperator' of node 'KubernetesPodOperator' is not configured…
el-aasi
  • 307
  • 2
  • 18
2
votes
1 answer

Airflow - KubernetesPodOperator - Role binding a service account

I am currently using the KubernetesPodOperator to run a Pod on a Kubernetes cluster. I am getting the below error: kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': '', 'Cache-Control':…
adan11
  • 647
  • 1
  • 7
  • 24
2
votes
2 answers

passing --serviceaccount in airflow kubernetes pod operator

I am trying to create and run a pod using Airflow kubernetes pod operator. The command below is tried and confirmed to be working and I am trying to replicate the same using the kubernetes pod operator locally kubectl run sparkairflow -n…
idk
  • 31
  • 1
  • 5
1
vote
1 answer

How to use pod_template_file parameter with KubernetesPodOperator in airflow 2?

I am running airflow via MWAA on aws and the worker nodes are running k8s. The pods are getting scheduled just fine but I am trying to use pod_template_file with KubernetesPodOperator, it's giving me a bunch of uncertain behavior. My template file…
Naxi
  • 1,504
  • 5
  • 33
  • 72
1
vote
1 answer

I am relatively new to Airflow and spark and want to use the Kubernetes Operator in airflow Dag to run a Spark Submit command

I am using the kubernetes version 1.25 client and server, I have deployed Airflow using the official helm charts on the environment. I want the Airflow dags kubernetes pod operator that has code to trigger the spark-submit operation to spawn the…
SAGE
  • 41
  • 5
1
vote
0 answers

Passing Airflow Context to init_container in KubernetesPodOperator

Airflow's KubernetesPodOperator provides an init_containers parameter, with which you can specify kubernetes init_containers. However init_containers expects a list of kubernetes.models.V1Container, and I don't see any way to pass airflow context…
ForeverWintr
  • 5,492
  • 2
  • 36
  • 65
1
vote
1 answer

KubernetesPodOperator on_failure_callback doesnt work on Airflow 2.4.2

When using the newest Airflow version (2.4.2), the on_failure_callback doesnt get triggered on KubernetesPodOperator tasks. The on_success_callback works just fine. For each KubernetesPodOperator task I set…
mato777
  • 11
  • 1
1
vote
0 answers

Accessing minikube via the KubernetesPodOperator in airflow

I am trying to make the airflow KubernetesPodOperator work with minikube. But unfortunately, the operator does not find the kubernetes cluster. The dag returned to me the following error: ERROR - HTTPSConnectionPool(host='192.168.49.2', port=8443):…
Elie Ladias
  • 111
  • 2
  • 16
1
vote
1 answer

KubernetesPodOperator is not able to access the Airflow Connections stored in Google Secrets Manager

I am using the Composer version 2.0.0 Airflow 2.1.4 and I have created a KubernetesPodOperator that is trying to access the Airflow connection stored in the Google Secrets Manager. But it isn't able to locate the credentials (check exceptions…
tank
  • 465
  • 8
  • 22
1
vote
0 answers

Accessing airflow KubernetesPodOperator environment variables

I am trying to pass environment variables to my container using env_vars parameters of the KubernetesPodOperator. task_fetch = KubernetesPodOperator( task_id=FETCH, name=FETCH, image=K8S_PRODUCTION_IMAGE, …
Elie Ladias
  • 111
  • 2
  • 16
1
2 3