How do I force a task on airflow to fail?

Question

I have a python callable process_csv_entries that processes csv file entries. I want my task to complete successfully only if all entries were processed successfully. Task should fail otherwise

def process_csv_entries(csv_file):
    # Boolean 
    file_completely_parsed = <call_to_module_to_parse_csv>
    return not file_completely_parsed

CSV_FILE=<Sets path to csv file>
t1 = PythonOperator(dag=dag,
                      task_id='parse_csv_completely',
                      python_operator=process_csv_entries,
                      op_args=[CSV_FILE])

t1 seems to complete successfully irrespective of returned value. How do I force PythonOperator task to fail?

score 52 · Accepted Answer · answered Mar 30 '17 at 10:51

52

raise exception when you meet the error condition ( in your case: when file is not sucesfully parsed)

raise ValueError('File not parsed completely/correctly')

raise relevant error type with suitable message

answered Mar 30 '17 at 10:51

Priyank Mehta

2,453
2
21
32

3

This works..Thanks! I was hoping there would be a better way to handle this. – Mask Mar 30 '17 at 20:38
16

This will not force the task instance status to "failed" though... Is there a way to bypass the retries config? – c-a May 25 '18 at 16:41

Alon Rolnik · Answer 2 · 2018-07-04T08:42:05.970

48

Yes, raise AirflowException, this will cause the task to move immediately to failure state.

from airflow import AirflowException

ValueError can be used for fail and retry.

edited Jul 04 '18 at 08:42

answered Jul 04 '18 at 06:15

Alon Rolnik

803
1
7
12

1

Any exception fails the task and moves it to failure state. – bcb Dec 14 '18 at 00:59
18

Is there a way to prevent airflow from retrying the task if retries are set? For example, there are some errors that you don't want/need to retry, such as invalid input related errors. – Joe J Apr 09 '19 at 17:08
1

@JoeJ, there is a PR open for that: https://github.com/apache/airflow/pull/7133 – gabra Mar 16 '20 at 17:35
AirflowException also avoids printing the stack trace on the logs +1 – Henrique Mendonça Jul 13 '23 at 09:01

score 25 · Answer 3 · edited Dec 19 '22 at 17:30

25

AirflowFailException is now available in Airflow 1.10.11 to make the task fail without retries

edited Dec 19 '22 at 17:30

yiwei

4,022
9
36
54

answered Aug 20 '20 at 18:10

y2k-shubham

10,183
11
55
131

5

For those of you wondering what version "now" means, it was [introduced in 1.10.11](https://airflow.apache.org/docs/apache-airflow/stable/changelog.html#airflow-1-10-11-2020-07-10) – ZaxR Jun 25 '21 at 15:30

officialrahulmandal · Answer 4 · 2021-05-01T05:29:19.617

23

if you want to fail the task without retries use AirflowFailException :-

Example :-

from airflow.exceptions import AirflowFailException
def task_to_fail():
    raise AirflowFailException("Our api key is bad!")

If you are looking for retries use AirflowException :-

Example:-

from airflow import AirflowException
def task_to_fail():
    raise AirflowException("Error msj")

edited May 01 '21 at 05:29

answered May 01 '21 at 05:18

officialrahulmandal

2,473
1
23
31

How do I force a task on airflow to fail?

4 Answers4