Emails with Airflow

Question

Do I have to do any additional setup in order to get apache-airflow to send me emails on task failure. I have the following in my config file (unchanged from the defaults):

[email]
email_backend = airflow.utils.email.send_email_smtp


[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
# smtp_user = airflow
# smtp_password = airflow
smtp_port = 25
smtp_mail_from = airflow@example.com

and in my task I have

'email': ['my.email@domain.co.uk'],
'email_on_failure': True,

But it is not sending me emails when the task fails.

I have seen this question: How do I setup Airflow's email configuration to send an email on errors? but I didn't really understand what actions to take from it. I'm just looking for the basics on what needs to be done before airflow will send emails. Do I have to configure my own smtp server or should it work out the box?

I'm running apache-airflow 1.9.0 on CentOS

score 7 · Accepted Answer · answered Aug 30 '18 at 13:31

7

You'll need your own SMTP service. We use Mailgun and Sendgrid internally and it works pretty well. You'll just need to change those lines in your .cfg with your credentials, restart airflow, and you should be good to go!

answered Aug 30 '18 at 13:31

Viraj Parekh

1,351
6
14

Thanks, do you have an opinion on what the easiest (free?) SMTP service to set-up is? I'll take a look at Mailgun and at Sendgrid now. Do I need both? – Dan Aug 30 '18 at 13:32
I believe Mailgun gives you 10K emails for free to start. Either one should work - you should only need one. – Viraj Parekh Aug 30 '18 at 13:36
I have set up an account with Mailgun and I have the [swaks example](https://documentation.mailgun.com/en/latest/quickstart-sending.html#send-via-smtp) from their site working in my terminal. Is their anything else I still need to do to get this working with airflow besides filling in my username and password? Must I change the `smtp_host` in the airflow config file to be `smtp.mailgun.org`? – Dan Aug 30 '18 at 16:02
2

Yeah I think you'll have to do that - along with the rest of the credentials. Shoudn't be anything outside of that – Viraj Parekh Aug 30 '18 at 19:42

score 2 · Answer 2 · answered Jan 14 '19 at 13:48

There is a Sendgrid subpackage available in airflow contrib if you prefer to use Sendgrid https://github.com/apache/airflow/blob/master/airflow/contrib/utils/sendgrid.py

And then just follow the instructions mentioned in the link.

    0. include sendgrid subpackage as part of your Airflow installation, e.g.,
    pip install apache-airflow[sendgrid] 

    1. update [email] backend in airflow.cfg, i.e.,
    [email]
    email_backend = airflow.contrib.utils.sendgrid.send_email

    2. configure Sendgrid specific environment variables at all Airflow instances:
    SENDGRID_MAIL_FROM={your-mail-from}
    SENDGRID_API_KEY={your-sendgrid-api-key}.

The email backend looks like it will be updated in Airflow 2.0 or thereabouts to `email_backend=airflow.providers.sendgrid.utils.emailer` as seen [here](https://github.com/apache/airflow/blob/master/airflow/providers/sendgrid/utils/emailer.py) — Colin Catlin, Jul 23 '20 at 21:50

Emails with Airflow

2 Answers2