Stack Overflow
Stack Overflow

Questions tagged [airflow-xcom]

21 questions
0
votes
1 answer

Apache airflow xcom for variable task id

I am currently working on a project which involves copying data from an S3 stage location to our tables managed by snowflake. However, it only works fine if I only have 1 table to copy data to(as I only need 1 task group and can therefore hard code…
Dynastywarriorlord07
  • 17
  • 3
0
votes
0 answers

Airflow failed to get task instance

i have tried to run a simple task using airflow bash operator but keep getting stuck on my DAG never stop running, it stays like green forever without success or fail, when i check the logs i see something like this. Thanks in advance for your time…
Efe
  • 1
  • 1
0
votes
1 answer

Airflow XCOMs communication from BashOperator to PythonOperator

I'm new to Apache Airflow and trying to write my first Dag which has a task based on another task (using ti.xcom_pull) PS : I run Airflow in WSL Ubuntu 20.04 using VScode. I created a task 1 (task_id = "get_datetime") that runs the "date" bash…
Salim Mzoughi
  • 41
  • 4
0
votes
1 answer

How to pass XCOM values to PythonVirtualenvOperator op_args?

So I am trying to pass arguments to op_args of PythonVirtualenvOperator using xcom_pull. But I am not quite sure how to do it properly. Any help would be highly appreciated. I am not sure how to provide more details, but stackoverflow doesn't allow…
Vlad Vlad
  • 530
  • 1
  • 5
  • 17
0
votes
1 answer

Using xcom pushed value as parameter in BigQueryToGCSOperator

I need to count the number of rows in a table and use the row count in the filename of an export to GCS. The following is an excerpt from my DAG. with models.DAG( 'my_dag', schedule_interval = '0 6 * * 1', start_date = datetime(2022, 1,…
Shahid Thaika
  • 2,133
  • 5
  • 23
  • 59
-1
votes
1 answer

Airflow ti.xcom_pull() return None

I have connection to local Postgres DB. I'm trying take some rows from table in 'task_get_titanic_data' with get_titanic_data function and apply some changes by second task task_process_titanic_data and process_titanic_data. import pandas as pd from…
Egor Ovchinnikov
  • 1
  • 1
1
2