Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
0
votes
1 answer

Airflow scheduled dags now working as expected

I cannot understand how scheduler works, I created some dags with: schedule_interval="0 21 * * *", start_date=pendulum.datetime(2022, 5, 3, tz="UTC"), catchup=False, "max_active_runs": 1, I activated the dag yesterday afternoon around 15.00 and I…
afmulone
  • 177
  • 1
  • 1
  • 8
0
votes
0 answers

Airflow DAG runs successfully but Tasks fail

I am trying to run a simple DAG on Airflow running on Docker. I've got two python scripts, the first one takes in the data using an API call, and the second one pushes the data into google sheets. So I've used t1 to execute the first python script…
Hamza Ahmed
  • 83
  • 6
0
votes
1 answer

How do I specify a bucket name using an s3 connection in Airflow?

We have an s3 bucket that Airflow uses as the source of all of our DAG data pipelines. We have a bucket for dev, test and production. Let's say the bucket in dev is called dev-data-bucket, in test it's called test-data-bucket etc. I don't want to…
Simon D
  • 5,730
  • 2
  • 17
  • 31
0
votes
1 answer

How to override the parameters of default args in a custom operator?

Hello I have a requirement where I have a custom operators in a dag and I want to override the parameter in one of the custom operator.But in custom operator @apply_defaults has been used.In this case if I override the value of a parameter of…
Ashutosh Rai
  • 123
  • 9
0
votes
1 answer

Airflow how to read JSON Input Params which includes '-' in the middle of param using Jinja

I’m running my DAG with configuration JSON, which includes parameters which includes -, e.g. market-name When I’m trying to read it using the following Jinja template: path_prefix = f"market={{{{ params.market-name }}}}/configuration" I’m getting…
Yaron
  • 10,166
  • 9
  • 45
  • 65
0
votes
2 answers

Airflow 2.0: Encapsulating DAG in class using Taskflow API

I have pipelines where the mechanics are always the same, a sequence of two tasks. So I try to abstract the construction of it through a parent abstract class (using TaskFlow API): from abc import ABC, abstractmethod from airflow.decorators import…
Nicolaz
  • 149
  • 1
  • 8
0
votes
2 answers

Import error ModuleNotFound for Airflow aws_lambda

I am using Airflow 2.2.5 and want to trigger an AWS lambda function. This is my Dockerfile FROM apache/airflow:latest RUN pip install --no-cache-dir apache-airflow-providers-docker==2.5.0 boto3==1.21.45 apache-airflow-providers-amazon==3.0.0 And…
dl.meteo
  • 1,658
  • 15
  • 25
0
votes
0 answers

Extract connections in airflow and use in boto3

I am. trying to extract connections from airflow as key,pass = BaseHook.get_connection('aws_default') print(conn.get_extra()) and use it in my boto3 connection as: account_id =…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

Triggering a DAG manually runs it on UTC schedule

I recently upgraded my airflow to 2.2.4. Now, from my admin console when I hit trigger DAG, it goes to queued. It does not immediately run but runs after 7 hours. The airflow scheduler and server are in the PDT timezone (7 hours behind UTC) so why…
Parth Shah
  • 1,237
  • 10
  • 24
0
votes
1 answer

Airflow:Slack Alert:Post the error message

Requirement: How to get the error message into the slack message Airflow version: 2.2.4 This below slack message only says it failed but doesn't say the error why it failed def slack_alert(context): SLACK_CONN_ID = 'slack_conn_id' …
Karthik
  • 441
  • 5
  • 17
0
votes
1 answer

why does my EMR terminate after last step, when started through Airflow?

I am using the code below to run EM, copy and execute a shell script. cluster_creator == Starting a cluster through EMR step_adder == Executing Shell scripts step_checker == Checking if steps are done. My question is, why does EMR terminate after…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

How to manage airflow pools slots?

I have the default pool with 128 slots. Now I have defined some other pools for each business_unit. A business_unit is a department, so the important data (prio 1) has the default pool available, but the prio 2 data has a pool for each…
mrc
  • 2,845
  • 8
  • 39
  • 73
0
votes
0 answers

Scheduler process per tag folder

I’m searching for some information about if is possible configure one scheduler processs of airflow per dag folder. I'm using separated dag folders depending on the business subject, and I need to know if is possible configure one scheduler process…
Bruno
  • 1
  • 2
0
votes
0 answers

Airflow 2 passing a task run-specific value throughout callbacks

I'm revising one of our company's custom operator in a way so that it logs its activity to a database. As part of that, I generate a unique number for use in initially populating the table row before the task runs, and then use that same number to…
RocketeerReporter
  • 1
  • 1
0
votes
1 answer

Poke the Specified Extension file in the server directory using the Airflow SFTPSensor

My use case is quite simple: When file dropped in the FTP server directory, SFTPSensor task picks the specified txt extension file and process the file content. path="/test_dir/sample.txt" this case is working. my requirement is to read the dynamic…
sri
  • 21
  • 3
1 2 3
…
45 46