2

I have been having issues with very slow unit test run throughs (2 hours +) in django with large data sets so wrote the following to enable faster testing, which got the time down to 40 mins by replacing all of our fixture loading tests with this which i initially wrote to replace our fixtures with static django model object creates.

from django.core.management import call_command
from django.db import transaction, DEFAULT_DB_ALIAS, connection, connections
from django import test
from django.test import TransactionTestCase
from django.test.testcases import disable_transaction_methods, \
    restore_transaction_methods


class FastTestCase(TransactionTestCase):
    """
    FastTestCase is a TransactionTestCase utilising a save point following the global setup
    to rollback following each test. This works in the same way FastFixtures would work,
    but allows rolling back to after the fixtures were loaded in order to speed up tests when
    very large data sets are being created.
    FastTestCase was written to assist in avoiding having to make a lot of changes to JSON fixtures
    when model changes had been made in order to enable unit tests to continue passing.
    NOTE: this is slower when small test data sets are being used!
    """

    @classmethod
    def setUpClass(cls):
        if test.testcases.connections_support_transactions():
            transaction.enter_transaction_management(using=DEFAULT_DB_ALIAS)
            transaction.managed(True, using=DEFAULT_DB_ALIAS)
            disable_transaction_methods()
            cls.fixtures_loaded = False

    @classmethod
    def tearDownClass(cls):
        if test.testcases.connections_support_transactions():
            restore_transaction_methods()
            transaction.rollback(using=DEFAULT_DB_ALIAS)
            transaction.leave_transaction_management(using=DEFAULT_DB_ALIAS)
        for connection in connections.all():
            connection.close()

    def globalSetUp(self):
        """
        Hook method to be overridden and provide fast fixtures style of setup
        """

    def _fixture_setup(self):
        from django.contrib.sites.models import Site
        Site.objects.clear_cache()

        if not self.__class__.fixtures_loaded:
            if hasattr(self, 'fixtures'):
                call_command('loaddata',*self.fixtures, **{'verbosity':0,'commit':False,'database':DEFAULT_DB_ALIAS})
            self.globalSetUp()
            if test.testcases.connections_support_transactions():
                self.__class__.fixtures_loaded = True
        if test.testcases.connections_support_transactions():
            self.save_point = transaction.savepoint()

    def _fixture_teardown(self):
        if test.testcases.connections_support_transactions():
            transaction.savepoint_rollback(self.save_point)
        else:
            super(FastTestCase, self)._fixture_teardown()

    def _post_teardown(self):
        self._fixture_teardown()
        self._urlconf_teardown()

I am now looking to improving the speed further is there any way i can make this faster & effective for smaller data sets?

stillwedge
  • 21
  • 3
  • Are you sure your tests are running in memory and not on any file-based database backend? 40min sounds very slow. – Cerin Jul 02 '14 at 22:16
  • We have a lot of custom datatypes which mean that the we are forced to use a mysql server in order to get production like results. I think the main performance hit that happened at the time of writing this was when a lot of new views were created and their unit tests more than doubled the run time. before that this Test case had Radically improved our Fixture based testing both in dev time if model changes were made and in the run time. Our main problem has always been that in order to test it properly the system needs a lot of data. – stillwedge Jul 03 '14 at 22:53

0 Answers0