intentionally raise an exception and fail the DAG

Question

In my Airflow dag, I am doing this:

                    try:
                        response = requests.get(url, params=params, headers=headers)
                        if response.status_code == 403:
                            raise Exception(f"Data Unavailable for metric: {m}, account: {account}")                          
                    except Exception as e:
                        logging.info(f"Exception during API request: {e} for metric: {m}, account: {account}")

The except Exception as e: part only runs when there's an error from the API for example a wrong API etc. However, in case of if response.status_code == 403, the DAG does not fail, even though I manually wrote a "raise Exception". It just moves on to the next steps and shows green or "success" for that task.

How can I force the DAG to fail, throw an error and stop if this error occurs?

Consider using [`response.raise_for_status`](https://requests.readthedocs.io/en/latest/api/#requests.Response.raise_for_status) to ensure that you fail for other non-200 response codes as well. — 0x5453, Apr 25 '23 at 13:49

ozs · Answer 1 · 2023-04-25T13:43:36.423

1

the "except Exception as e:" comes after you raise the exception so it catch it and no exception actually raised in the task. also notice you don't raise after the logging in the Exception section which swallow the exception.

   try:
        response = requests.get(url, params=params, headers=headers)
        response.raise_for_status()
   except HTTPError as e:
       if e.response.status_code == 403:
           raise Exception(f"Data Unavailable for metric: {m}, account: {account}")
   except Exception as e:
       logging.info(f"Exception during API request: {e} for metric: {m}, account: {account}")
       raise e

edited Apr 25 '23 at 13:43

answered Apr 25 '23 at 08:05

ozs

3,051
1
10
19

Why is no exception raised? The exception does raise successfully in other cases (for example when the api key is incorrect). Even if i completely remove the **if statement**, there is no exception raised because the 403 code is not considered an "error". How can I ensure that an exception is raised in both cases? – x89 Apr 25 '23 at 08:17
it raise but it inside the "try" so the except section catch it (I am sure the log is printed). the practice is to raise specific Errors and this way you can handle it different. – ozs Apr 25 '23 at 08:56
yes, the logs are printed but even with your code, it doesn't throw an error. I want the DAG's task to fail and end process so that that next dag doesn't run. How can I do that? – x89 Apr 25 '23 at 11:50
I mean the next step of the dag (next task) should not work when this one fails – x89 Apr 25 '23 at 12:27
you need to raise error at the Exception section. – ozs Apr 25 '23 at 13:34

intentionally raise an exception and fail the DAG

1 Answers1