Django Unittests Failing Randomly

Question

I am running into this wierd issue of randomly failing unittests. I am basically testing Syndication Feeds

Here is the relevant portion of my code:

class ObjectFeedTests(PuppyTestCase):

    def test_object_feeds(self):
        site = Site.objects.get_or_create(id=site_id)[0]

        #Some preprocessing and initialization

        with self.settings(ROOT_URLCONF='myapp.urls.something'):

            #Some more preprocessing is done here

            show = ShowFactory.create(slug=show_slug, title=show_name,
                                      site=site)

            for feed_name, entry_points in OBJECT_FEEDS.items():

                factory = default_factories[model] #fetches the correct factory

                if feed_name == "podcast":
                    instance = factory.create(show=show, slug="podcasts")
                else:
                    instance = factory.create()

                #Creates the instance with the factory correctly

                for feed_type in ['rss', 'atom']:

                    with self.settings(SITE_ID=site.id):
                        class_url_pattern = feed_class().get_url_pattern()

                    slug_of_note = '%s/%s' % (show_slug, instance.slug)
                    presumed_url = '/%s/%s/%s' % (FEED_URL_PREFIXES[feed_type],
                                           feed_name, slug_of_note)

                    with self.settings(SITE_ID=site.id):
                        response = self.client.get(presumed_url,
                                        {'password': '****'})

                    self.assertContains(response, '<?xml',
                        msg_prefix="%s %s feed raised a 404 \
                        or wasn't XML (tried %s)" \
                            % (feed_name, feed_type, presumed_url))

                # End of our loop.  We'll delete the object
                # so that we're not hung up by unique_together constraints
                instance.delete()

The factory:

class ShowFactory(factory.Factory):
    FACTORY_FOR = Show

    title = factory.Sequence(lambda name: 'My Show {0}'.format(name))
    slug = factory.Sequence(lambda name: 'my-show-{0}'.format(name))
    creator = factory.SubFactory(UserFactory)
    last_editor = factory.SubFactory(UserFactory)
    site = factory.SubFactory(SiteFactory)
    status = STATUS_CHOICES.PUBLISHED

Each time I run the test, 3 factories are created, and the tests fail randomly.

If I have left out any local variable declaration in the code here, please ignore it, as I have condensed the code to keep it as small as possible.

I was considering using this, but the results were still the same.

Please let me know if there is any more information I could provide.

Here is the stacktrace of a failure:

======================================================================
FAIL: test_object_feeds (tests.syndication_tests.ObjectFeedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/cases.py", line 44, in run
    testMethod()
  File "tests/syndication_tests.py", line 370, in test_object_feeds
    presumed_url))
  File "/usr/local/virtualenvs/lib/python2.6/site-packages/django/test/testcases.py", line 622, in assertContains
    " (expected %d)" % (response.status_code, status_code))
AssertionError: 404 != 200 : people rss feed raised a 404                             or wasn't XML (tried /feeds/show/test-show-0): Couldn't retrieve content: Response code was 404 (expected 200)

----------------------------------------------------------------------

Ran 5 tests in 10.613s

FAILED (failures=1)
Destroying test database for alias 'default'...

The next time I run it,

======================================================================
FAIL: test_object_feeds (tests.syndication_tests.ObjectFeedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/cases.py", line 44, in run
    testMethod()
  File "tests/syndication_tests.py", line 370, in test_object_feeds
    presumed_url))
  File "/usr/local/virtualenvs/lib/python2.6/site-packages/django/test/testcases.py", line 622, in assertContains
    " (expected %d)" % (response.status_code, status_code))
AssertionError: 404 != 200 : channels rss feed raised a 404                             or wasn't XML (tried /feeds/show/test-show-1): Couldn't retrieve content: Response code was 404 (expected 200)

----------------------------------------------------------------------
Ran 5 tests in 3.483s

FAILED (failures=1)
Destroying test database for alias 'default'...

Stupid question, but what does the test say in it's output? Why is it failing? — Henrik Andersson, Jun 27 '13 at 15:55
For some reason the thing you're getting isn't valid XML, which is pretty obvious. Your test is looking good and I can't see any reason as to why it wont work. — Henrik Andersson, Jun 27 '13 at 16:04
The test is throwing a response code of 404, instead of 200. That is when it fails. The funny thing is, it throws a 404 on a different object every time i run the tests. — karthikr, Jun 27 '13 at 16:07
I am working on changing the way I load fixtures to make sure it's not the data — karthikr, Jun 28 '13 at 09:09

score 3 · Accepted Answer · answered Jul 15 '13 at 23:29

3

Please forgive me for stating the obvious. The most common cause for random failure is that one of your tests did not tear down properly. This happens randomly because the order in which the tests are executed is non-deterministic. Try running the failed test alone instead of running the whole test suite. If it passes every time, then it is definitely caused by some other tests.

Another possibility is that you may be reusing the old test fixture depending on the arguments you provide to run your tests

answered Jul 15 '13 at 23:29

Conan Li

474
2
6

Thanks for your response. Can you elaborate on the second point on what you mean ? – karthikr Jul 15 '13 at 23:38
This will depend on your test runner. For example, I am using django-nose, which allow me to do REUSE_DB=1 ./manage.py test and save several seconds at the beginning and end of your test suite by reusing the test database from the last run. – Conan Li Jul 15 '13 at 23:57
Thank you. I guess your direction on the fixtures made me think I might have some data issue in the fixtures. I shall try that out – karthikr Jul 19 '13 at 15:51
As it turns out fixing the fixtures made me move forward. – karthikr Aug 05 '13 at 19:36
@karthikr what did you exactly do with your fixtures to fix this? – András Gyömrey Oct 11 '17 at 13:09
If i remember correctly I had a few fixtures that were inconsistent. Had a few required fields missing which were being picked up in some joins and were not on some causing assertions to fail – karthikr Oct 11 '17 at 20:12

Django Unittests Failing Randomly

1 Answers1