I am having trouble configuring Airflow to automatically send error messages to Sentry.
I have set sentry_on and sentry_dsn in environment variables (AIRFLOW__SENTRY__SENTRY_ON and AIRFLOW__SENTRY__SENTRY_DSN), and these appear in the Airflow GUI Admin > Configuration display. From the Sentry docummentation (https://docs.sentry.io/platforms/python/guides/airflow/) this should be enough to send error messages to Sentry on exceptions.
But when I deliberately throw an exception (example below) no error is sent to Sentry.
(I am confident that the sentry_dsn value works, as our code has custom code that we use in failure callbacks that does send error messages to Sentry.)
Reading the Airflow and sentry-python code, it seems that exceptions when running a task should automatically be sent to Sentry because TaskInstance._run_raw_task is decorated with Sentry.enrich_errors. enrich_errors calls sentry_sdk.capture_exception which ultimately sends the error message to Sentry.
But I am not sure that Sentry.enrich_errors is being called. I have written code patching it (also below) to log the call, but when the exception is thrown nothing is logged. (Perhaps there is something wrong with the patching code? Similar code patching TaskInstance._run_raw_task did result in log messages.)
Here is my code deliberately raising an Exception and patching Sentry.enrich_errors:
import logging
from datetime import datetime, timedelta
from functools import wraps
from airflow import DAG
from airflow.configuration import conf
from airflow.exceptions import AirflowException
from airflow.operators.python import PythonOperator
from airflow.sentry import Sentry
# Verify that the sentry_on configuration variable is set to True
logging.info("sentry_on is %s" % str(conf.getboolean("sentry", "sentry_on", fallback=False)))
class MySentry:
def enrich_errors(self, func):
""" Patch for enrich_errors to log if it has been called. """
@wraps(func)
def wrapper(_self, *args, **kwargs):
logging.info("MySentry.enrich_errors has been called")
return func(_self, *args, **kwargs)
return wrapper
# Sentry.enrich_errors = MySentry.enrich_errors
def throw_exception_function():
raise Exception("still another exception thrown on purpose, no DAG or Task on_failure_callback, patch enrich_errors")
with DAG(
dag_id="deliberate_exception__patch_enrich_errors",
default_args={
"owner": "DAG owner",
"depends_on_past": False,
"start_date": datetime(2017, 1, 1),
"retries": 0,
"retry_delay": timedelta(minutes=0),
},
description="Deliberately throws exception without on_failure_callback for DAG or task; patches enrich_errors",
schedule_interval=None,
catchup=False,
) as dag:
Sentry.enrich_errors = MySentry.enrich_errors
exception_task = PythonOperator(
task_id="exception_task",
python_callable=throw_exception_function,
)
Please note that this code does not send errors to Sentry when the patching is commented out.
Why isn't my configuration automatically sending error messages to Sentry?
Why isn't Sentry.enrich_errors being called?
Any help would be appreciated.
