Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
2
votes
1 answer

Airflow - run DAG at start of data interval

I would like to create a monthly dag that runs at the start of each month and which uses a logical date as the start of the same month (so e.g. March’s logical run date would actually run on March 1 instead of waiting until the end of the period…
PratipP
  • 21
  • 3
2
votes
1 answer

airflow.exceptions.AirflowException: Dag could not be found; either it does not exist or it failed to parse

I recently upgraded the Airflow from 1.10.11 to 2.2.3 after following the steps given in https://airflow.apache.org/docs/apache-airflow/stable/upgrading-from-1-10/index.html. I first up upgraded to 1.10.15 as suggested which worked fine. But after…
Cherry
  • 31
  • 5
2
votes
1 answer

How I can use Airflow template reference in the DAG python code

I am new in the Airflow world and trying to understand one thing. For example I have a DAG that contains 2 tasks. The first task is submitting spark job, and the second one is Sensor that waits for a file in s3. RUN_DATE_ARG =…
David Shpilfoygel
  • 23
  • 4
2
votes
0 answers

How to run DAG from code ? (without getting message: No run dates were found for the given dates and dag interval)

I'm trying to debug airflow simple code (from pycharm). I set executor = DebugExecutor in airflow.cfg I wrote simple code: from airflow import DAG from airflow.operators.python import PythonOperator from airflow.utils.dates …
user3668129
  • 4,318
  • 6
  • 45
  • 87
2
votes
1 answer

how can I stop the [UtcDateTime cache_ok ]warn? When I run airflow dag

when I run airflow server : My version is airflow lastest(2.2).when I run the program,the dag log "SAWarning: TypeDecorator UtcDateTime(timezone=True) will not produce a cache key because the ``cache_ok`` attribute is not set to True. This can have…
Join Michael
  • 25
  • 3
2
votes
0 answers

Airflow imports into tasks

I'm wondering, if Airflow refreshes dags every X time, How it is better (performance speaking) to import modules that will be used inside tasks: A: def _task_function(ti): import json ... B: import json def _task_function(ti): .... And…
Daniel Fridman
  • 167
  • 9
2
votes
0 answers

Airflow DAG Dependencies / Orchestration between different workflows

My team is working orchestrating our data pipeline with Airflow. Since our pipeline steps are complex, we were thinking about having different DAGs / workflows, each defined on its own file. Each of the workflows can trigger more than one downstream…
Raquel
  • 89
  • 8
2
votes
0 answers

apache airflow ExternalTaskMarker clear another dag's task recursively but task state is None

i'm testing ExternalTaskSensor and ExternalTaskMarker. ExternalTaskSensor wait until external Dag's Task finished and ExternalTaskMarker clear another dag's task…
PSW
  • 59
  • 6
2
votes
1 answer

Airflow SFTPToS3Operator - download multiple files from STFP to S3

I am new to Airflow. My requirement is to develop a data pipeline to do below: i) Connect and download multiple csv files (filename starting with name) airflow from given SFTP folder to S3 ii) Ingest files from S3 to redshift I have used…
jasmeet
  • 21
  • 1
2
votes
1 answer

Facing Issue with DataprocCreateClusterOperator (Airflow 2.0)

I'm trying to migrate from airflow 1.10 to Airflow 2 which has a change of name for some operators which includes - DataprocClusterCreateOperator. Here is an extract of the code. from airflow.providers.google.cloud.operators.dataproc import…
codninja0908
  • 497
  • 8
  • 29
2
votes
0 answers

How can I disable internal Airflow errors to be logged via sentry but still se DAG execution errors?

I have enabled sentry [1.5.0] with apache airflow [Version 2.2.2] using following configuration in config file. [sentry] sentry_on: True sentry_dsn: [DSN] sentry_environment: dev It seems to work just fine for all sort of errors. However Airflow…
JD-V
  • 3,336
  • 1
  • 17
  • 20
2
votes
0 answers

Airflow 2.2.0 / Flask-Appbuilder / Task exited with return code Negsignal.SIGKILL

After upgrading from Airflow 2.1.4 to 2.2.0 (or 2.2.1 or 2.2.2), we have a number of DAGs that fail to execute. The error Airflow provides is "Negsignal.SIGKILL". This happens when running locally with the SequentialExecutor AND when we deploy to…
William Patterson
  • 51
  • 2
2
votes
1 answer

Monitoring progress of a long running task in Apache Airflow 2

I'm trying to move an adhoc-controlled and monitored workflow to Airflow 2. The workflow consists of multiple steps, quite a typical use case, with a single exception - one step is a very long-running job. This job might take from a few minutes to a…
dreo
  • 910
  • 1
  • 9
  • 20
2
votes
0 answers

How to trigger a task in Airflow when it's parent task fails, but skip it when it's in upstream_failed?

As I can see there is no Trigger Rule what can differentiate failed and upstream_failed statuses. We have to behave differently when the parent has run but failed and when it has not run at all. When the status of the parent task is: sucecess ->…
elaspog
  • 1,635
  • 3
  • 21
  • 51
2
votes
1 answer

set priority for the multiple dag runs

I have a dag that I want to run multiple times say 30. But airflow can parallelly execute 16 dag runs at a time. Suppose one dag run takes longer time to execute , then among the 14 dag runs waiting , I want to run 10th dag run first. Is there any…
ansh
  • 21
  • 1
1 2 3
…
45 46