Loading SQL dump before running Django tests

Question

I have a fairly complex Django project which makes it hard/impossible to use fixtures for loading data.

What I would like to do is to load a database dump from the production database server after all tables has bene created by the testrunner and before the actual tests start running.

I've tried various "magic" in MyTestCase.setUp(), but with no luck.

Any suggestions would be most welcome. Thanks.

If I'm not mistaken loading sql would be faster because it doesn't have the overhead that fixtures do. I'm looking to solve this same problem. I have a large DB to load for testing and I'd like to keep the loading quick. — Jeff, Dec 30 '09 at 20:58
I use generic relations extensively which is a problem when using fixtures. It appears this has just been solved in the work towards 1.2, see http://docs.djangoproject.com/en/dev/topics/serialization/#natural-keys — knutin, Dec 31 '09 at 18:51
It's a shame you can only vote comments up, not down. That first comment just reeks. — boatcoder, Oct 25 '12 at 21:38
@S.Lott - here's an example. I'm trying to add tests to a legacy project that was never written with fixtures in mind. Loading fixtures triggers lots of unwanted behaviour. I'll eventually fix this problem - but I need tests in place before I start a big refactor. And I need data to run the tests against. — Andy Baker, Aug 28 '15 at 17:10

score 8 · Answer 1 · answered Apr 07 '10 at 01:39

Django supports loading SQL files when doing syncdb, reset, or starting a test runner -- this does exactly what you describe:

http://docs.djangoproject.com/en/dev/howto/initial-data/#providing-initial-sql-data

You need to create an "sql" directory in your app directory, and then put a file named "mymodel.sql" in that directory (where "MyModel" is the corresponding model name).

myproject/
   |--myapp/
       |--sql/
           |--mymodel.sql

You can create this SQL with dump tools for your database.

SQLite [1]: echo '.dump' | sqlite3 yourdbname.sqlite > myapp/sql/mymodel.sql
MySQL [2]: mysqldump yourdbname > myapp/sql/mymodel.sql
PostgreSQL [3]: pg_dump yourdbname > myapp/sql/mymodel.sql

After dumping, you'll need to edit the file to remove everything but the appropriate INSERT statements, or other complicated stuff. In particular, you must remove transaction handling, index creating, and table creating SQL to avoid errors when loading duplicate create statements.

I use this method for loading really, really big fixtures -- it takes far too long to process the json, but a straight sql import is pretty snappy.

Do be aware that this method will load the sql for any invocation of synchdb, reset, etc. in addition to loading data for the test runner -- so you won't be able to have different data for different test cases, and you'd have to remove files before a reset if you didn't want them loading back onto your production server.

[1] http://www.sqlite.org/sqlite.html

[2] http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html

[3] http://www.postgresql.org/docs/8.1/static/backup.html#BACKUP-DUMP

Since this answer was posted, this behavior has become unavailable in Django. Django 1.2 now ignores `sql` files in the `sql/` directory when running the test framework. — Trey Hunner, Mar 06 '11 at 01:40

score 1 · Accepted Answer · edited Jul 07 '18 at 20:38

You may need to look into defining a custom test runner. There's some info here: https://docs.djangoproject.com/en/dev/topics/testing/advanced/#other-testing-frameworks

Basically I think you can just copy the default test runner from django.test.simple.run_tests and then modify it to suit your needs.

I've not done this before, but from my understanding that would be the way to customize this.

score -1 · Answer 3 · answered Dec 30 '09 at 14:14

Fixtures are the best option. Have you tried using ./manage.py dumpdata to create a fixture from your current database? I have not seen that fail on complex models, but I guess it's possible.

Assuming you're using mysql, you should be able to script this by using mysqldump.

Loading SQL dump before running Django tests

3 Answers3

Linked