Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
0
votes
1 answer

Setting multiple DAG dependency in Airflow

I have multiple Ingestion DAGs -> 'DAG IngD1', 'DAG IngD2', 'DAG IngD3' , and so on which ingest data for individual tables. After the ingestion DAGs are completed successfully, I want to run a single transformation DAG -> 'DAG Tran'. Which means…
aslfkhsalfgals
  • 13
  • 2
0
votes
1 answer

Airflow Xcom not getting resolved return task_instance string

I am facing an odd issue with xcom_pull where it is always returning back a xcom_pull string "{{ task_instance.xcom_pull(dag_id = 'cf_test',task_ids='get_config_val',key='http_con_id') }}" My requirement is simple I have pushed an xcom using python…
Divyaansh Bajpai
  • 232
  • 1
  • 8
0
votes
1 answer

Unable to get epoch in milliseconds in Airflow

I have the below airflow code: time_for_run_id = time.time() run_id=int(time_for_run_id * 1000) SQL_1 = "INSERT INTO dds_control.dds_control_table (job_run_id,job_start,job_status)…
aslfkhsalfgals
  • 13
  • 2
0
votes
0 answers

Airflow externalTaskSensors to sense multiple dags with different schedule

I am trying to create dependency between multiple dags. Lets say Dag_A, Dab_B and and running every day at 14:15 and 14:30 respectively. now i want to run Dag_C which runs at 14:30 having 2 sensors ( ExternalTaskSensors) each for above dags. I am…
yahoo
  • 183
  • 3
  • 22
0
votes
1 answer

Problem with BranchPythonOperator to TaskGroup

I got the following DAG import logging from airflow import DAG from airflow.operators.dummy import DummyOperator from airflow.operators.python import BranchPythonOperator from airflow.utils.dates import days_ago from airflow.utils.task_group import…
tristobal
  • 426
  • 2
  • 7
  • 18
0
votes
1 answer

Apache Airflow 2.3.0: Dags are not appearing in the browser

Can any one please help me to resolve this error, that is appearing in the Airflow 2.3.0 console. Do not use SQLite as metadata DB in production – it should only be used for dev/testing. We recommend using Postgres or MySQL. Click here for more…
Nikitha Singh
  • 11
  • 5
0
votes
1 answer

How to hide aws_key_id and aws_secret_key under Rendered Template in AWS MWAA

I am using AWS Managed Airflow (MWAA) with Airflow version 2.0.2. To setup External Tables in Snowflake I am using IAM User and I need to pass aws_key_id and aws_secret_key in Create Stage statements. I have templated my Create Stage statements and…
Mohan Hegde
  • 11
  • 4
0
votes
1 answer

How do I set up a postgres connection on airflow for the postgres operator

For the rtfm crowd, let me document my suffering. I went here: https://betterdatascience.com/apache-airflow-postgres-database/ But my ui has UNAUTHORIZED in pink after I add the info. I also went…
Henrietta Martingale
  • 771
  • 8
  • 27
0
votes
1 answer

An error occurred (403) when calling the HeadObject operation: Forbidden in airflow (2.0.0)+

Error - *** Failed to verify remote log exists s3://airflow_test/airflow-logs/demo/task1/2022-05-13T18:20:45.561269+00:00/1.log. An error occurred (403) when calling the HeadObject operation: Forbidden Dockerfile - FROM apache/airflow:2.2.3 COPY…
Aditya Sinha
  • 614
  • 8
  • 18
0
votes
0 answers

Why Airflow is raising an SLA?

I don’t understand why SLA’s sometimes are triggered. An example is a dag with an SLA of 'sla': timedelta(hours=1) I received an email with this: Here’s a list of tasks that missed their SLAs: APP_task_group.copy_events on…
mrc
  • 2,845
  • 8
  • 39
  • 73
0
votes
1 answer

Airflow 2.1.0 - DAG 'XXXXXXXX' not found in serialized_dag table

We recently upgraded from Airflow 1.10.12 to Airflow 2.1.0 and are getting issues errors like this in the UI for each of our dags. We have upgraded the version from 1.10.12 to 1.10.15 and it was successful. Then from 1.10.15 we have upgraded to…
Amal Bose Mathew
  • 1
  • 1
  • 1
0
votes
2 answers

Airflow Only works with the Celery, CeleryKubernetes or Kubernetes executors

I got this dag, nevrtheless when trying to run it, it stacks on Queued run. When i then trying to run manually i get error: Error: Only works with the Celery, CeleryKubernetes or Kubernetes executors Code: from airflow import DAG from…
Arie
  • 3,041
  • 7
  • 32
  • 63
0
votes
1 answer

Docker-compose and extended Dokerfile: External Providers not found

I've deployed apache airflow using docker-compose using an extended image to include requirements. I can see in the airflow UI that the snowflake providers package was installed (apache-airflow-providers-snowflake), but for some reason, I'm getting…
pa4enko11
  • 76
  • 1
  • 8
0
votes
1 answer

Is it safe to delete airflow.db file?

I can't login in the web UI of airflow. I always get this Invalid login error. When I run the airflow users list the users are there. So my option is to delete the airflow.db and run airflow db init again. I want to know if it is safe to do that?
mee
  • 688
  • 8
  • 18
0
votes
1 answer

Airflow 2.0.2 - Hourly DAG getting stuck seeing Refreshing TaskInstance repeatedly

I've been noticing that some of the DAG runs for an hourly DAG are being skipped, I checked the log for the DAG run before it started skipping and noticed it had actually been running for 7 hours which is why other DAG runs didn't happen, it is…
KristiLuna
  • 1,601
  • 2
  • 18
  • 52
1 2 3
…
45 46