Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
0
votes
0 answers

Airflow - Test a TaskGroup

Is there a way to test all tasks within a TaskGroup in Airflow without running each task separately? I have multiple task groups within a dag. I don't want to test the whole dag. I also don't want to have to test the tasks individually. See below…
j7skov
  • 557
  • 7
  • 22
0
votes
2 answers

How to get maximum/minimum duration of all DagRun instances in Airflow?

Is there a way to find the maximum/minimum or even an average duration of all DagRun instances in Airflow? - That is all dagruns from all dags not just one single dag. I can't find anywhere to do this on the UI or even a page with a…
pragmatic learner
  • 419
  • 3
  • 12
0
votes
1 answer

How can get the execution hour in Airflow DAG context

I would like to get the execution hour inside a DAG context. I checked and found that {{ds}} provides only the execution date and not time. Is there any way to get the hour at which the DAG gets executed on any given day ? with…
RTM
  • 759
  • 2
  • 9
  • 22
0
votes
2 answers

DAG level global static variable for all task instances

I am trying to share a common variable with all the tasks e.g pipeline_id which i calculate using the current_system time. Is there a way to pass this variable to all the tasks in DAG? currently i get different value in different tasks as they are…
Varun Sharma
  • 39
  • 2
0
votes
2 answers

Airflow try number issue

I have encountered an error during testing , does anyone have any lead on this. Code: relative_path = self._render_filename(ti, ti.try_number) Error: AttributeError: 'str' object has no attribute 'try_number
jonuchauhan
  • 1
  • 1
0
votes
1 answer

Airflow XComArg object has no attribute output

I am trying to pull output from an Airflow task and dynamically spin up multiple task instances based on the previous task output. But when I try to read output from filter_backfill_info as filter_backfill_info.output i get XComArg object has no…
pyhotshot
  • 445
  • 8
  • 23
0
votes
1 answer

airflow: templated argument with BigQueryCheckOperator

I would like to use the BigQueryCheckOperator in airflow where the query needs to have the execution date as parameter. I have written the following operator that check the current date, but I do not understand how to use the execution_date of the…
Galuoises
  • 2,630
  • 24
  • 30
0
votes
1 answer

Airflow:SnowflakeOperator:Last query id returned to XCOM

I have a snowflake file with a query like as below, in the snowflake operator if I have a return so that I can pass xcom to the next task. How can I get only the last query id to be returned for xcom ? Basically I need to get the snowflake last…
Kar
  • 790
  • 13
  • 36
0
votes
1 answer

Airflow 2.3 - Changing a task instance state as failed without raising

Using task flow, let's say I have: from airflow.utils.state import State from airflow.operators.python import get_current_context @dag( schedule_interval=None, start_date=datetime(2021, 1, 1) ) def myDag(): @task() def getData(): …
Flo
  • 377
  • 2
  • 15
0
votes
1 answer

Passing task outputs with AirFlow XCOM

I have an SSHOperator that writes a filepath to stdout. I'd like to get the os.path.basename of that filepath so that I can pass it as a parameter to my next task (which is an sftp pull). The idea is to download a remote file into the current…
inspectorG4dget
  • 110,290
  • 27
  • 149
  • 241
0
votes
1 answer

Apache Airflow (MWAA) DAG that can send a file from PythonOperator to SSHOperator

I'm looking to create a DAG that can run two tasks. The ideal workflow would be to create a PEM key file then SSH using that newly created file. One task would be with the PythonOperator to call AWS Secrets Manager through the SecretsManagerHook.…
rk92
  • 551
  • 4
  • 21
0
votes
0 answers

Airflow:Passing of XCOM value to next task for each multiple tasks

Requirement: Pass XCOM value from the previous task to the next tasks for each multiple tasks Error: Error rendering template: can only concatenate str (not "set") to str I tried as below params={"s3_key": list(date.keys())[0]} Reading the params…
Kar
  • 790
  • 13
  • 36
0
votes
2 answers

How to redirect a user to the airflow homepage after authentication?

I am trying to use my own SSO for airflow web. After I authenticate and authorize a person, I am lost on how to redirect this authenticated user to the airflow homepage to access the dags. I found no direct code or understanding on how to do this.…
Wei Wei Ryu
  • 3
  • 1
0
votes
1 answer

Loop in Airflow 2.x Task not recognizing task

What im trying to do is that i want to run a BigQueryCreateEmptyTableOperator x number of times, it depends on how many files are in bucket (one for every file). For filling the list, I use…
Aldo
  • 1
  • 1
0
votes
1 answer

Airflow & Google Drive API - only able to use folders within My Drive, not shared Team folders

When attempting to use the GoogleDriveToGCSOperator via a Service Account, I am only able to interact with folders I create directly within the My Drive path in Drive. If I use a different Team/Shared Drive, I receive errors even though the Team…
phenderbender
  • 625
  • 2
  • 8
  • 18
1 2 3
…
45 46