Airbrake notifications disabled in Django tests?

Question

I've configured a Django app called lucy_web to log errors to Airbrake using pybrake. In a module in the lucy_web hierarchy, lucy_web.lib.session_recommendations, I've defined a testing function:

import logging

logger = logging.getLogger(__name__)

def log_something():
    logger.error("Logging something...")

If I call this function from the Django shell, like so:

(venv) Kurts-MacBook-Pro-2:lucy-web kurtpeek$ python manage.py shell
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from lucy_web.lib.session_recommendation import *

In [2]: log_something()

I see an error email appear:

However, if I define a test, and try to call it from there, I don't see any new instances of the error appear:

from django.test import TestCase
from django.core import mail


class SessionRecommendationTestCase(TestCase):
    def test_airbrake_notification_if_session_type_does_not_exist(self):
        from lucy_web.lib.session_recommendation import log_something
        log_something()
        import ipdb; ipdb.set_trace()

Running this did not cause any new instances of the error appear in the Airbrake dashboard. Also, mail.outbox is empty:

(venv) Kurts-MacBook-Pro-2:lucy-web kurtpeek$ python manage.py test lucy_web.tests.test_session_recommendation.SessionRecommendationTestCase.test_airbrake_notification_if_session_type_does_not_exist
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
--Return--
None
> /Users/kurtpeek/Documents/Dev/lucy2/lucy-web/lucy_web/tests/test_session_recommendation.py(377)test_airbrake_notification_if_session_type_does_not_exist()
    375         from lucy_web.lib.session_recommendation import log_something
    376         log_something()
--> 377         import ipdb; ipdb.set_trace()

ipdb> mail.outbox
[]

As I understand from https://docs.djangoproject.com/en/2.0/topics/testing/tools/#email-services, within a test, Django redirects all email sent by Django to a dummy outbox, mail.outbox. I would not expect this to the case for Airbrake, though, since the emails are probably sent by their backend - and if it were the case, I wouldn't expect mail.outbox to be empty.

Why am I not getting any Airbrake notification for the test? Is it something more general, like that all logging is disabled/captured during the test?

score 0 · Accepted Answer · answered Jun 01 '18 at 22:06

I verified by setting a trace in the the pybrake.Notifier's send_notice method that it does not get called in the unit test, whereas it does when I call the function 'manually' from the Django shell.

To still get test coverage, I worked around this problem simply asserting that logger.error gets called:

from django.test import TestCase
from unittest.mock import patch

class SessionRecommendationTestCase(TestCase):
    @patch('lucy_web.lib.session_recommendation.logger.error')
    def test_create_session_from_non_existent_session_type_title_triggers_error_logging(self, mock_error):
        title = "Being Awesome"    # Title of a non-existent session type
        _create_sessions([title])
        mock_error.assert_called_with(
            f"Tried to create a session of type '{title}', but no such session type was found in the database.")

where this is the actual function I'm trying to test:

def _create_sessions(title_list, starting_number=1, **kwargs):
    '''Auxiliary function to create sessions from a list of SessionType titles'''
    session_number = starting_number
    for title in title_list:
        session_type = SessionType.objects.filter(title=title).first()
        if session_type:
            Session.objects.create(
                session_type=session_type,
                session_number=session_number,
                **kwargs)
            session_number += 1
        else:
            logger.error(
                f"Tried to create a session of type "
                f"'{title}', but no such session type was found in the database.")

This test passes:

(venv) Kurts-MacBook-Pro-2:lucy-web kurtpeek$ python manage.py test lucy_web.tests.test_session_recommendation.SessionRecommendationTestCase.test_create_session_from_non_existent_session_type_title_triggers_error_logging
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK
Destroying test database for alias 'default'...

Airbrake notifications disabled in Django tests?

1 Answers1