1

I have a DAG defined that contains a number of tasks, the last of which is only run if any of the previous tasks fail. This task simply posts to a Slack channel that the DAG run experienced errors.

What I would really like is if the message sent to the Slack channel contained the actual error that is logged in the task logs, to provide immediate context to the error and perhaps save Ops from having to dig through the logs.

Is this at all possible?

Cam
  • 2,026
  • 3
  • 25
  • 42
  • That´s totally possible and this is ideally done using the Xcom -interface, see [this](https://stackoverflow.com/questions/38574151/airflow-pass-parameters-to-dependent-task) question or the [airflow-example](https://github.com/apache/airflow/blob/master/airflow/example_dags/example_xcom.py). – dorvak Mar 05 '19 at 17:54
  • While this is possible with `Xcom`s, the implementation comes with challenges. To be able to gracefully push the error message, you'll have to wrap `execute()` method of your `operator`s into a `try..except` block and push (or just `return` it) `str(exception)` to `Xcom`. This will however interfere if have to publish an `Xcom` [by returning value](https://github.com/apache/airflow/blob/master/airflow/example_dags/example_xcom.py#L42) (other than the exception itself) from `execute()` method. I'd suggest you write an `operator` / `mixin` that could add this behaviour to any other `operator` – y2k-shubham Mar 06 '19 at 00:45
  • In fact I believe since `XCom`s are the only way of inter-`task` communication *provided by* `Airflow`, it is your *only resort* – y2k-shubham Mar 06 '19 at 00:47

0 Answers0