Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
2
votes
1 answer

How to support Airflow jinja template in sqlfluff

I want to run sqlfluff fix command to this sql file. This SQL is for Snowflake and will be executed in Airflow. ti.xcom_pull is a function used in Airflow DAG. DELETE FROM test WHERE date = '{{ti.xcom_pull(key='filedate')}}'; then I got this…
Yohei Onishi
  • 1,362
  • 1
  • 19
  • 42
2
votes
1 answer

airflow KeyError: TaskInstanceKey

running airflow 2.2.5 on python3.6, I get this error after running a dag from console: airflow dags test airflow_report1_email 2022-08-30 ... File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 70, in wrapper …
martin
  • 862
  • 9
  • 28
2
votes
1 answer

Dag which can read CSV rows as an input for operators

I have a csv file with various columns of data that can be utilized for Python functions called by my PythonOperators. My dags pipeline is set up in such a way that I want to read the CSV for each row and feed those inputs into my operators. But how…
Danish Iqbal
  • 43
  • 6
2
votes
1 answer

Airflow 2 Error sending Celery task: Timeout

I am in the process of migrating our Airflow environment from version 1.10.15 to 2.3.3. I have migrated 1 DAG over to the new environment and intermittently I get an email with this error: Executor reports task instance finished (failed) although…
wymangr
  • 189
  • 3
  • 16
2
votes
1 answer

Airflow UI failing to fetch same-task different-retry logs from different nodes in cluster

I kind-of "inherited" a project that uses Airflow 2.2.4 installed on a cluster of several nodes (meaning that I wasn't part of the deployment decisions and configurations and I might not be aware of some under-the-hood processes). Each node runs a…
Giulia Savorgnan
  • 63
  • 6
2
votes
1 answer

Airflow logging doesn't show up when log from inside operator

I'm trying to log from inside an function in a dag, but it hasn't been working. I tried using print("something") as well but that didn't print any out either in the airflow log. How should I do logging here? Many thanks for your help. import…
user3735871
  • 527
  • 2
  • 14
  • 31
2
votes
2 answers

airflow 2 helm chart - how to specify mysql connection string

I'm deploying Airflow 2 on GKE Autopilot using helm chart and have provisioned a Cloud SQL instance (MySQL) to be used as DB by airflow. I have created (using kubectl) a secret in K8s with this connection string as value and wanted to give that as…
Rohit Shisode
  • 142
  • 8
2
votes
1 answer

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: Ident authentication failed for user "airflow"

I have a CentOS 7 instance,and I've installed airflow and postgresql on top of that. Initially installed the postgres-server and the contrib via yum by following the steps…
Kulasangar
  • 9,046
  • 5
  • 51
  • 82
2
votes
0 answers

Make Airflow custom operator compliant with Taskflow API

I just discovered the Taskflow API by watching the Airflow summit 2022. It works fine, but I still heavily rely on "classic" operators, some of them I have written myself. Is there documentation that explains how I could make my custom operators…
Thomas J
  • 146
  • 10
2
votes
0 answers

How can I download a file from an S3 bucket after I SSH to a remote machine/edge node to a particular path in the same remote machine?

I am relatively new to Airflow and am trying to solve a particular problem. I wanted to know how I could download my file residing in an S3 bucket after SSH to a remote machine/client through the SSHOperator. Can I simply use some bash script (I…
Danish Ansari
  • 21
  • 1
2
votes
1 answer

How to send multiple emails using EmailOperator in Airflow

I'm trying to send individual emails to a list of users using an EmailOperator inside a python loop but currently the emails are not being sent. The dag doesn't return any error which makes me believe the EmailOperator is not being…
Shadow Walker
  • 979
  • 5
  • 27
  • 51
2
votes
1 answer

Is it possible to get multiple values from XCOM for a single task at once in Airflow?

Is it possible to retrieve multiple values from XCOM, pushed by a single task but with different keys? I think I've seen examples to this: # pulls one value pulled_value = ti.xcom_pull(key=None, task_ids='my_task_id') and to this: # pulls multiple…
elaspog
  • 1,635
  • 3
  • 21
  • 51
2
votes
1 answer

Reading XCOM and Airflow variables probably slows down Airflow (in Google Cloud Composer)

We are trying to merge daily (CSV) extract files into our Data Warehouse. In our use case the DAG's python code is the same for all of our DAGs (~2000), so we generate them by a DAG generator logic from a single python file. In our DAGs we only have…
elaspog
  • 1,635
  • 3
  • 21
  • 51
2
votes
1 answer

How to configure remote logging for airflow?

I successfully deployed Airflow 2.2.4 to GCP Kubernetes Engine. However, I have an issue with logging. I am using the official helm chart. So the, thing is, I defined a connection ID using the Airflow UI and the connection ID works for tasks as they…
Marin
  • 861
  • 1
  • 11
  • 27
2
votes
2 answers

Unable to access newly created Airflow UI MWAA

I am trying to create MWAA as root user and I have all AWS services (s3 and EMR )in North California. MWAA doesn't exist in North California. Hence created this in Oregon. I am creating this in a private network, it also required a new s3 bucket in…
Xi12
  • 939
  • 2
  • 14
  • 27
1 2 3
…
45 46