Stack Overflow
Stack Overflow

Questions tagged [airflow]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.

Airflow is a workflow scheduler. It was developed by Airbnb to manage its complicated workflows.

References

Related Tags###

Similar workflow schedulers:

10104 questions
3
votes
1 answer

Airflow Openshift installation with Dockerfile

I tried to install the Airflow via my own image at a public dockerhub, but it works perfect locally, but when I tried to use it on Openshift. I got this error bellow. `ERROR: Could not install packages due to an OSError: [Errno 13] Permission…
Bruno Justino Praciano
  • 512
  • 9
  • 27
3
votes
1 answer

Airflow schedule_interval and start_date to get it to always fire the next interval

How can I configure airflow (mwaa) so that it will fire at the same time (6am PST) every day regards of when the dag is deployed? I have tried what makes sense to me: set the schedule_interval to 0 6 * * *. set the start date to: now =…
cosbor11
  • 14,709
  • 10
  • 54
  • 69
3
votes
0 answers

airflow GoogleCloudStorageObjectSensor with wildcard not working file*.xml

I wanted to use filemask in the GoogleCloudStoragePrefixSensor. I cant use the GoogleCloudStoragePrefixSensor because I also need to see the ending oif the file mask. BAsically my file is like "tv_link_input_*.xml". So, tried using…
Mohammed
  • 43
  • 5
3
votes
1 answer

Airflow UI Changing Execution Datetime to Readable Format

In Airflow's UI, if I hover over any of my task IDs, it'll show me the "Run", "Started", and "Ended" dates all with a very verbose format i.e. 2021-02-12T18:57:45.314249+00:00. How do I change the default preferences in Airflow's UI so that it…
kashmoney
  • 412
  • 1
  • 5
  • 17
3
votes
2 answers

How to run cloud composer task which loads data into other project BigQuery Table

I have my cloud composer environment created under project-A and I want to load data into other project-B BigQuery table. I know about task its GCSToBigQueryOperator but its not succeeding its failing, I want to know how can I achieve this. From…
user3065757
  • 475
  • 1
  • 5
  • 14
3
votes
1 answer

Airflow on Docker give an error "ModuleNotFoundError: No module named 'google' "

I'm trying use airflow on Docker. my_python.py file in dags directory like : from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime, timedelta import argparse import psycopg2 import csv import…
EEks
  • 51
  • 5
3
votes
2 answers

passing parameters to the UI when triggering a dag on airflow

I'm trying to find a way to generalize passing parameters of a dag when triggering it from the UI itself,I know I should pass it as a key/value pair format but I don't know how to parse those parameters in the dag script itself for example if I pass…
omar
  • 41
  • 1
  • 4
3
votes
1 answer

Can Apache airflow be used on Windows? What should be done in advance?

I know Apache airflow doesn't support Windows. But I know I can also use windows using wsl. The main question is: Can I get selenium crawling or api through Apache airflow without using wsl? If #1 is impossible, is it possible using wsl?? If use…
skw0314
  • 107
  • 7
3
votes
1 answer

Run k8s commands within Airflow hosted on an EKS cluster using the python api

I currently have an Airflow deployment hosted on an EKS cluster, and want it to run a report that will check the logging for another deployment and alert me if any errors have occurred. Locally I'm able to run this without issue as I can just point…
davo777
  • 286
  • 2
  • 15
3
votes
2 answers

Airflow 2.0 Docker setup

Recently been trying to learn Airflow, but a majority of resources online depended on this repo https://github.com/puckel/docker-airflow which unfortunately has been removed. I am not familiar with docker so I'm just trying to set up locally and…
kjay
  • 383
  • 3
  • 16
3
votes
1 answer

Airflow DAG serialization: TypeError: Object of type 'V1Pod' is not JSON serializable

"When using the KubernetesExecutor, Airflow offers the ability to override system defaults on a per-task basis. To utilize this functionality, we can create a Kubernetes V1pod object and fill in the desired overrides." I am trying to trigger a DAG…
Pascal GILLET
  • 135
  • 1
  • 14
3
votes
2 answers

on_failure_callback not working in airflow DAG()

I want to call two different functions for dag failure and success. For that i want to use on_failure_callback and on_success_callback in DAG() function. As per my requirement, this callbacks should be on dag level and not task level. That's why i…
Joseph D
  • 189
  • 1
  • 12
3
votes
1 answer

Using Airflow, does the MsSqlOperator accept responses from SQL Server?

I have a SQL Operator that creates a simple json. The end goal is that json being sent to a rest API. I'm finding the process of sending a HTTP POST in SQL code complicated, so if I can get the json kicked back to airflow I can handle it from there.…
Tevett Goad
  • 138
  • 1
  • 7
3
votes
1 answer

Get context from Pod launched with Airflow KubernetesPodOperator

We have some dags that launch pods using KubernetesPodOperator and I'm trying to get some information inside the pod, like dag_id, task_id, try_number, environment, etc. I know that I can get this information from the context of the Airflow task…
strider
  • 683
  • 9
  • 25
3
votes
1 answer

How can I add new "Conn Types" to Airflow 2.0?

I'm learning Apache Airflow 2.0 and I was creating a new "Connection" but in "Conn Type" there are just a few options, for example I want to add a MySQL connection but there isn't that option. In this post (Apache Airflow - Connection issue to MS…
Alex Vargas
  • 53
  • 1
  • 5
1 2 3
…
99
100