Stack Overflow
Stack Overflow

Questions tagged [airflow-api]

77 questions
0
votes
1 answer

Hitting the Airflow API running on Astronomer cloud

We currently have airflow DAGs running on an Astronomer cloud instance, and we’re trying to develop external functions that would eventually trigger these DAGs. Referencing the documentation for this → Make Requests to the Airflow REST API |…
dlsupaolo11
  • 23
  • 3
0
votes
2 answers

Airflow 2: Check if a task still exists or not with task_id

we have created a task for sensor operation, but the task name will be dynamically updated. i.e., f"{table_name}_s3_exists". We have a scenario where we have to check a table's location twice, but if the task is still present, we don't have to…
Santhosh
  • 71
  • 1
  • 5
0
votes
2 answers

How to get reason for failure using slack in airflow2.0

How to get the reason for the failure of an operator, without going into logs. As I want to post the reason as a notification through slack? Thanks, Xi
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

How to get exact reason for failure in EMR through slack?

I need to have a slack notification when my pyspark code/ bootstrap fails with the exact reason for failure in EMR. I am using airflow 2.0, and I am using on_failure_callback=task_fail_slack_alert to notify slack. But it is not notifying. Is my…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

branch trigger rule all_success or none_failed does not work, task gets executed even if it is not supposed to

I have an airflow DAG as below task1 -> task2 -> [task3_1, task3_2] task3_1 -> task4_1 -> task5_1 task3_2 -> task4_2 -> task5_2 [task5_1,task5_2] -> task_6 -> task_7 -> task_8 Tasks 1,2,6,7,8 are linear with 2 parallel jobs for 3,4,5 in between. I…
Venkatesh Gotimukul
  • 741
  • 1
  • 9
  • 30
0
votes
1 answer

Azure service bus trigger does not raise error message when calling a bad request to a URL and does not fail the job

My service bus queue trigger will get triggered every time when a new file is landed in blob storage, and the trigger function will perform a post request to Airflow to trigger an airflow job, but doesn't matter if it is a good request with 200 or…
Ling Xu
  • 1
  • 1
0
votes
1 answer

401 From MWAA Airflow Environment When Attempting To Run A DAG

Description When attempting to make calls to a Managed Amazon Airflow instance I was not able to make API calls to the airflow environment despite being able to generate a CLI token from aws mwaa create-cli-token. Why am I getting a FORBIDDEN error…
AlexLordThorsen
  • 8,057
  • 5
  • 48
  • 103
0
votes
0 answers

Concurrent Airflow DAGS

Could anyone please let me know whether I can spin up many (say, around 10 concurrent DAGs) concurrent DAGs simultaneously in parallel at the same time? The parameters for these DAGs will be the same but with different values. Any ideas or…
testbg testbg
  • 193
  • 2
  • 11
0
votes
1 answer

After I externally trigger the airflow dag using experimental api it keeps on running and not executing the dag

Airflow Dag: def print_hello(**kwargs): task_params = kwargs['dag_run'].conf['task_payload'] print('Hello world a with {}'.format(task_params)) def hello_world(): args = { "start_date": datetime(2022, 5, 1), "retries":…
kishan agarwal
  • 115
  • 2
  • 5
0
votes
1 answer

communication between top level python code and operator variable in airflow

i am writing airflow code where i am running one python operator to read user input given from cli and want to use result return by that operator in top level python code for dynamic task generation . using Xcom we can perform inter task…
Yug
  • 105
  • 1
  • 9
0
votes
1 answer

AirFlow sends exception wrapped with HTTP 200 OK

When we call a DAG using REST, the AirFlow tries to execute it but if there is an exception it sends back the stack trace as a response body wrapped with HTTP 200 OK. How can we change that feature so that if there is any execution error, it would…
Sujoy
  • 802
  • 11
  • 22
0
votes
1 answer

Which Airflow permissions allow triggering of a DAG Run through the API?

Using Airflow 2.0.2, I'm trying to use the airflow API to trigger DAG Runs. When I run a simple GET like curl -X GET --user "fooUser:passw0rd" "${ENDPOINT_URL}/api/v1/pools" I get expected results: { "pools": [ { "name":…
Mike S
  • 1,451
  • 1
  • 16
  • 34
0
votes
0 answers

DAG not running simultaneously

We have one DAG which used to work pretty well. We later automated the process and ran the same DAG file using different threads 4-8 times almost simultaneously. It is throwing an error that says it is duplicate. How to make it unique so that all…
Sujoy
  • 802
  • 11
  • 22
0
votes
1 answer

airflow dag getting triggered more than once

I have a dag with one task, and I want it to get triggered only once a day. My problem is that it gets triggered multiple times when the time comes. So the daily task is run 4 times instead of once. I set a number of configurations to fix that…
asal
  • 1
  • 1
0
votes
1 answer

context.log_url as href in airflow not working

I do not want to use Slack. I 'm just using custom notification(on_failure_callback) function in args for my mailing function. I have used def task_my_alert(context): task_id=context['task'].task_id dag_run = context.get('dag_run') …
Sudarshan
  • 13
  • 5
1 2 3 4
5
6