Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
0
votes
0 answers

Dags triggered by TagDriggenRunOperator are only queued but not running

I have a parent dag that is used to trigger another. When I trigger the child dag, the child dags is only queued but never running. This is my parent_dag.py file from airflow import DAG from datetime import datetime, timedelta from…
shubh gupta
  • 94
  • 7
0
votes
1 answer

How to use Docker Airflow - ExternalPythonOperator - python=os.fspath(sys.executable)?

GOAL I need this because a ExternalPythonOperator feature available since 19 OCT. 2022. = Airflow 2.4.0 https://airflow.apache.org/docs/docker-stack/build.html#important-notes-for-the-base-images ExternalPythonOperator -…
sogu
  • 2,738
  • 5
  • 31
  • 90
0
votes
0 answers

Airflow invoking dynamic tasks sequentially

I am trying to invoke two dynamic mapped tasks sequentially i.e fetch_work_retry needs to be invoked only after process_work is completed. Below is the code which I am using. With this fetch_work_retry gets invoked before process_work is complete.…
Arul
  • 143
  • 3
  • 12
0
votes
1 answer

Pull list xcoms in TaskGroups not working

My airflow code has the below Python Operator callable where I am creating a list and pushing it to xcoms: keys = [] values = [] def attribute_count_check(e_run_id,**context): job_run_id = int(e_run_id) da = "select count (distinct…
djgcp
  • 163
  • 1
  • 14
0
votes
0 answers

Decoding messages from airflow PubSub Operators to BigQuery

I'm using an airflow 2 PubSubPublishMessageOperator in order to pass a message from an xcom.pull to a bigquery tables using the latest direct BigQuery Subscription https://cloud.google.com/pubsub/docs/bigquery send_to_pubsub =…
Snowfire
  • 171
  • 9
0
votes
1 answer

"The requested operation cannot be performed because there is a file transfer in progress" error when doing multiple SFTP operation

We try to download the file from sftp and then delete the file (via Airflow), we get keeping this error on one of the sftp servers, it's working fine on other sftp servers. [Errno 13] The requested operation cannot be performed because there is a…
Reza
  • 18,865
  • 13
  • 88
  • 163
0
votes
1 answer

Airflow - Task flow, Dynamic Task Mapping with multiple output

Let's say I have an Airflow (2.3) DAG that looks like this: @task def retrieve_ingest_setup_queries(): settings = Variable.get("athena_settings", deserialize_json=True) # settings = {'drops': ["DROP TABLE my_table", "DROP TABLE my_table2"], …
Flo
  • 377
  • 2
  • 15
0
votes
1 answer

Airflow Docker AWS EC2 DAG file log error

ISSUE I have run other dag files previously they all give this message even if they pass or fail. Goal Get this error fixed Log that contains the error I got this after running simple dag 241adsgf1108 *** Log file does not exist:…
sogu
  • 2,738
  • 5
  • 31
  • 90
0
votes
0 answers

How can I use a xcom value to configure max_active_tis_per_dag for the TriggerDagRunOperator in Airflow 2.3.x?

Dear Apache Airflow experts, I am currently trying to make the parallel execution of Apache Airflow 2.3.x DAGs configurable via the DAG run config. When executing below code the DAG creates two tasks - for the sake of my question it does not matter…
Pantonaut
  • 191
  • 1
  • 8
0
votes
1 answer

Recursively copy s3 objects from one s3 prefix to another in airflow

I am trying to copy files that I receive hourly into my incoming bucket with the below…
Gladiator
  • 354
  • 3
  • 19
0
votes
0 answers

How make airflow output line number in dga file during error?

Consider the error: Broken DAG: [/usr/local/airflow/dags/mydag.py] Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 840, in dag dag.add_task(self) File…
Cherry
  • 31,309
  • 66
  • 224
  • 364
0
votes
1 answer

Apache Airflow min_file_process_interval and DAGs schedule interval

The concepts of Apache Airflow state that there should not be any expensive database calls in the top-level code of DAGs, because .py files are parsed every min_file_process_interval seconds. On the other hand, I read that your DAGs (or workflow)…
montty
  • 11
  • 1
  • 3
0
votes
1 answer

Using Airflows S3Hook is there a way to copy objects between buckets with different connection ids?

I'm copying files from an external companies bucket, they've sent me an access key/secret that I've set up as an env variable. I want to be able to copy objects from their bucket, I've used the below but that's for moving objects with the same…
KristiLuna
  • 1,601
  • 2
  • 18
  • 52
0
votes
1 answer

How to enter password during runtime in Airflow

I have a script (RegisterKerbTicketAndOAuth.sh file) in remote cluster, I am connecting to remote cluster from Airflow using ssh operator and logged in successfully which is successfull Now the problem is, when I run the sh file like this sh…
L2607
  • 33
  • 3
0
votes
0 answers

Airflow dag automatically Triggers

I have created the dag with the following configuration job_type='daily' SOURCE_PATH='/home/ubuntu/daily_data' with DAG( dag_id="transformer_daily_v1", is_paused_upon_creation=False, default_args=default_args, …
Ritik Gupta
  • 11
  • 2
1 2 3
…
45 46