Stack Overflow
Stack Overflow

Questions tagged [kubernetesexecutor]

21 questions
6
votes
4 answers

Airflow- dag_id could not be found issue when using kubernetes executor

I am using airflow stable helm chart and using Kubernetes Executor, new pod is being scheduled for dag but its failing with dag_id could not be found issue. I am using git-sync to get dags. Below is the error and kubernetes config values. Can…
user2416
  • 223
  • 4
  • 18
5
votes
4 answers

Airflow Unittest.cfg permission problems?

So, I'm setting up Airflow on EKS. Everything seems to be in place, except that I cant use the KubernetesExecutor. Everytime I try to use it, I get this error: with open(TEST_CONFIG_FILE, 'w') as f: PermissionError: [Errno 13] Permission denied:…
JaMo
  • 85
  • 6
3
votes
2 answers

airflow error "dag_id could not be found" when running airflow on KubernetesExecutor

I am using this helm chart to deploy airflow https://github.com/apache/airflow/tree/master/chart Apache Airflow version: 2.0.0 Kubernetes version: v1.19.4 What happened: I get this error when try to execute tasks using kubernetes [2021-01-14…
suliman
  • 334
  • 6
  • 17
3
votes
1 answer

Airflow doesn't recognise my S3 Connection setting

I am using Airflow with Kubernetes executor and testing out locally (using minikube), While I was able to get it up and running, I cant seem to store my logs in S3. I have tried all solutions that are described and I am still getting the following…
midNight
  • 155
  • 1
  • 4
  • 18
2
votes
1 answer

Airflow installation with helm on kubernetes cluster is failing with db migration pod

Error: Steps: I have downloaded the helm chart from here https://github.com/apache/airflow/releases/tag/helm-chart/1.8.0 (Under Assets, Source code zip). Added following extra params to default values.yaml, createUserJob: useHelmHooks:…
veeresh patil
  • 1,168
  • 1
  • 11
  • 18
2
votes
2 answers

Assign Gitlab Runner daemon's pod and the jobs's pods to two separate node groups in Kubernetes when using Kubernetes executor

We're using Gitlab Runner with Kubernetes executor and we were thinking about what I think is currently not possible. We want to assign the Gitlab Runner daemon's pod to a specific node group's worker with instance type X and the jobs' pods to a…
Rshad Zhran
  • 496
  • 4
  • 17
2
votes
0 answers

User "system:serviceaccount:gitlab:default" cannot get resource "deployments" in API group "apps" in the namespace "gitlab"

I am trying to run a k8s deployment job from gitlab kubernetes executor. I deployed the kubernetes runner using helm as following. my values.yaml includes the following rbac rules: rbac: create: true rules: - apiGroups: [""] resources:…
sqr
  • 365
  • 2
  • 12
  • 29
2
votes
0 answers

Use Helm GitLab-Runner in Kubernetes for Gitlab and Docker Registry with self signed certificate (x509 Error)

Hei guys, i have set up a local kubernetes cluster with k3s (currently 1 Intel NUC) using ubuntu server 18 LTS. I also have a working Synology Diskstation with GitLab and a private Registry (running on docker). Both (GitLab and Registry) are using…
Simon
  • 31
  • 4
2
votes
1 answer

Does Airflow Kubernetes Executor run any operator?

I am assessing the migration of my current Airflow deployment from Celery executor to Kubernetes (K8s) executor to leverage the dynamic allocation of resources and the isolation of tasks provided by pods. It is clear to me that we can use the native…
André Perez
  • 50
  • 5
2
votes
0 answers

Unable to run Airflow DAGs with KubernetesExecutor - AirflowException: dag_id could not be found

I am new to Airflow and have been trying for a while now to get a task ran using the KubernetesPodOperator class without any luck. The pod gets created and synchronizes the DAGs from a git repo, I don't see any errors retrieving the DAGs but the pod…
Sebastián Lopez
  • 21
  • 4
1
vote
0 answers

Airflow logs are empty in UI

I have a DAG that ran for 2 hours but I can not see any log in the UI. In addition, I can't see how many attempts there were (or if there were any at all). In the airflow webserver pod (where the UI runs), I can see some logs, but they are not…
simple_developer
  • 43
  • 5
1
vote
1 answer

Airflow KubernetesExecutor / Failed to adopt pod

We are using the airflow kubernetes executor and for the most part it works great. While normally pods get terminated and disappear after a completed task, sometimes "something" happens and these completed pods end up sticking around forever. Or…
William Patterson
  • 51
  • 2
1
vote
2 answers

Airflow 2.0.1: Pod Template Override not working as expected for KubernetesExecutor

Setup: Airflow 2.0.1 with Kubernetes 1.18 and Python 3.8, Kubernetes Client: 18.17.x Pod template file: apiVersion: v1 kind: Pod metadata: name: workerPod spec: containers: - args: [] command: [] env: - name: …
spuri
  • 177
  • 1
  • 2
  • 9
1
vote
2 answers

Airflow KubernetesExecutor, logs do not show up in UI until after executor pods complete

I have started using KubernetesExecutor and I have set up a PV/PVC with an AWS EFS to store logs for my dags. I am also using s3 remote logging. All the logging is working perfectly fine after a dag completes. However, I want to be able to see the…
user6407048
  • 181
  • 1
  • 3
  • 11
1
vote
3 answers

pod_mutation_hook function not working on airflow running in kubernetes using KubernetesExecutor

I am attempting to migrate an airflow deployment running in kubernetes from the CeleryExecutor to the KubernetesExecutor. Everything went smoothly in my local development environment (running on minikube), however I need to load a sidecar container…
WillySchu
  • 166
  • 8
1
2