Django Testing: no data in temporary database file

Question

I'm using a sqlite3 database set up as follows in settings.py:

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.spatialite',
        'NAME': 'path/to/config.sqlite',
        'TEST_NAME': 'path/to/test-config.sqlite',
        # ... USER, PASSWORD and PORT left out for brevity
    }
}

During a test run started with:

python manage.py test myapp.mytest

this temporarily creates a database file path/to/test-config.sqlite which I need in another application loaded with the required fixtures.

The database file however is empty, which I asserted during a pause in one test:

sqlite> select * from someapp_somemodel;

... no results here :(

Other test cases which do not require a sqlite file and for which the in-memory database suffices, no errors occurs.

My questions:

• Why doesn't django flush its data to the database file if it creates it anyway? and
• How can I convince django to do it, as I require the data to be dumped in the temporary database file?

EDIT

I'm using Django 1.3.1, if thats of any interest.

EDIT2

I'm familiar with fixtures, and I use them to populate the database, but my problem is that the data from the fixtures is not written to the database file during the test. Sorry if I wasn't clear enough on that fact.

EDIT3

As my question needs some clarification, please consider the following test setup (which is close to what I'm actually doing):

class SomeTestCase(django.test.TestCase):
    fixtures = ["some_fixture.json", "some_other_fixture.json"]

    def testSomething(self):
        import pdb; pdb.set_trace()

When the testSomething method runs into the breakpoint I start up the sqlite3 program and connect to the temporary database file created by Django. The fixtures are loaded (which I know, because other tests work aswell) but the data is not written to the temporary database file.

Answer 1

Perhaps it has to do with how Django tests use transactions?

Check if the issue persists if you turn off all transactions in settings.py:

DISABLE_TRANSACTION_MANAGEMENT = True

Answer 2

Have you run the initial syncdb to create the database tables? (python yourproject/manage.py syncdb)

In your settings.py, what apps do you have installed under INSTALLED_APPS?

In your project, what models have you built?

Depending on what apps you have installed in INSTALLED_APPS and what custom models you have added to your project, will dictate what databases syncdb will create.

Answer 3

From the django documentation:

...When using the SQLite database engine the tests will by default use an in-memory database (i.e., the database will be created in memory, bypassing the filesystem entirely!). If you want to use a different database name, specify TEST_NAME in the dictionary for any given database in DATABASES.

Furthermore, if you read about fixture loading:

...At the start of each test case, before setUp() is run, Django will flush the database, returning the database to the state it was in directly after syncdb was called...

If you need a fresh database with some fixtures loaded you can of course do so by creating an empty database with running syncdb and load some fixtures with django-admin.py loaddata!

Answer 4

It looks like your are doing something with GeoDjango based on your database engine, 'django.contrib.gis.db.backends.spatialite'. GeoDjango needs extra setup and so my guess is it needs something extra for testing too. (If you aren't using GeoDjango just use 'django.db.backends.sqlite3' as your database engine. Keep the TEST_NAME though otherwise Django will create the test DB in memory.)

There is a whole process for loading in the spatial data and the GeoDjango installation docs are very thorough. Here's a link direct to creating the SpatiaLite database. You'll need to do basically the same thing for testing.

From the testing docs:

You will need to download the initialization SQL script for SpatiaLite:

$ wget http://www.gaia-gis.it/spatialite/init_spatialite-2.3.zip

$ unzip init_spatialite-2.3.zip

If init_spatialite-2.3.sql is in the same path as your project's manage.py, then all you have to do is: $ python manage.py test

Answer 5

I found a way, but as I also messed with hdparm (with -F or -W 0/1), i don't know if it will work for you. I did reboot and re-try to make sure thought. Also, this test does not have SpatiaLite but as you said this is probably irrelevant.

Anyway, we need 2 screens to reproduce this, screen0 runs the tests and screen1 is an sh shell to work with will screen0's process is paused.

Start the test (screen0):

>>> ./manage.py test testapp
Creating test database for alias 'default'...
Destroying old test database 'default'...
Type 'yes' if you would like to try deleting the test database 'db_test.sqlite', or 'no' to cancel: yes
--Return--
None
> /home/jpic/testproject/testapp/tests.py(16)testSomething()
     14 
     15     def testSomething(self):
---> 16         import ipdb; ipdb.set_trace()

Check the size of the created test database file, (screen1):

<<< 18:00.39 Mon Feb 20 2012!~/testproject 
<<< jpic@germaine!10004 env
>>> ls -l db_test.sqlite
-rw-r--r-- 1 jpic jpic 49152 2012-02-20 18:00 db_test.sqlite

Run PRAGMA SYNCHRONOUS sql command from python (screen0):

ipdb> from django.db import connection; cursor = connection.cursor()
ipdb> cursor.execute("PRAGMA SYNCHRONOUS")
<django.db.backends.sqlite3.base.SQLiteCursorWrapper object at 0x294f348>
ipdb> 

Check if the size of the database file increased (screen1):

<<< 18:00.42 Mon Feb 20 2012!~/testproject 
<<< jpic@germaine!10005 env
>>> ls -l db_test.sqlite
-rw-r--r-- 1 jpic jpic 272384 2012-02-20 18:00 db_test.sqlite

Data has been written to the file.

It doesn't make much sense to me because apparently PRAGMA SYNCHRONOUS alone should just query for the value (in my case: 2/FULL). But in practicee, it writes to disk. Note that if you haven't 2 (FULL) then you should set it to 2: PRAGMA SYNCHRONOUS 2.

Now, I can't say what was written (was it fully written ?) because I can't get my hands on the test database: if i run sqlite db_test.sqlite in screen1 to get a client on the test database: I can't run any command (nor a select, nor a .dump) because "SQL error: database is locked". But, I guess that's your problem now B)