Django data migration fails when running manage.py test, but not when running manage.py migrate

Question

I have a Django 1.7 migration that looks something like this:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations

def units_to_m2m(apps, schema_editor):
    Interval = apps.get_model("myapp", "Interval")
    IntervalUnit = apps.get_model("myapp", "IntervalUnit")

    for interval in Interval.objects.all():
        IntervalUnit(
            interval=interval,
            unit=interval.unit,
            base_date=interval.base_date
        ).save()

class Migration(migrations.Migration):

    dependencies = [
        ('otherapp', '0007_auto_20150310_1400'),
        ('myapp', '0009_auto_20150316_1608'),
    ]

    operations = [
        migrations.CreateModel(
            name='IntervalUnit',
            # ...
        ),
        # ...
        migrations.AddField(
            model_name='interval',
            name='units',
            field=models.ManyToManyField(to='otherapp.Unit', through='myapp.IntervalUnit'),
            preserve_default=True,
        ),
        migrations.RunPython(units_to_m2m),
        migrations.RemoveField(
            model_name='interval',
            name='unit',
        ),
        migrations.RemoveField(
            model_name='interval',
            name='base_date',
        ),
    ]

When I run manage.py migrate, it migrates just fine. When I run manage.py test, however, it tries to create the test database, then fails in the middle of this migration with the following error:

Traceback (most recent call last):
...
  File "/home/adam/myproject/myapp/migrations/0010_auto_20150317_1516.py", line 10, in units_to_m2m
    for interval in Interval.objects.all():
...
django.db.utils.OperationalError: (1054, "Unknown column 'myapp_interval.base_date' in 'field list'")

When I connect to the test database afterwards (it doesn't delete it), the database structure looks as you'd expect after the migration has run, even though it crashed half way through. What's going on here?

Edit: I've tried splitting the migration into three separate migrations, one containing all the stuff before the RunPython one, one containing RunPython on its own, and one containing all the stuff that's afterwards; it's still doing the exact same thing.

The migrations should be ok until it reaches "migrations.RunPython(units_to_m2m)". It seems that the "scheduling" model is not ready yet for that operation in 'myapp_interval.base_date'. I would try to put that model as a dependency and see what the outcome is. — brunofitas, Mar 19 '15 at 10:00
'scheduling' and 'myapp' are one and the same, that's just me failing to anonymise the code properly :S — Daisy Leigh Brenecki, Mar 19 '15 at 10:13

score 4 · Answer 1 · answered Jul 23 '15 at 21:51

This is kind of weird and we don't know why this works, but we changed our allow_migrate signature in our router to the following:

def allow_migrate(self, db, app_label, **hints):
    """
    Make sure the mydb db does not allow migrations
    """
    if db == 'mydb':
        return False

    return True

And this error mysteriously went away. Note that this signature does not match what is in the 1.8 documentation (we're using 1.8.2) allow_migrate(db, app_label, model_name=None, **hints) as shown here: https://docs.djangoproject.com/en/1.8/topics/db/multi-db/#allow_migrate

But hopefully this will help you?

I don't understand why this works either, but it solved the same problem for me. Thanks. — shacker, Oct 20 '16 at 22:43

score 3 · Accepted Answer · answered Mar 20 '15 at 00:22

It turns out, the migrations were running successfully, in the order they were supposed to, but I have two databases, and it was running my migrations on both without consulting the database router. The Django ticket to track this problem is #23273, which is still open.

Presumably, the RunPython migration was querying against default (which had already been migrated) rather than the database the migration was actually supposed to run on.

In my case, we no longer needed to use the second database for anything, so we were able to remove it from settings.DATABASES altogether.

score 0 · Answer 3 · answered Jun 29 '21 at 20:13

I had the same issue on Django 3.1.5. I initially had disabled the test migration for the 1st app using the MIGRATION_MODULES flag in the setting file.

MIGRATION_MODULES = {
     'app1': None
}

I ended up solving the test issue by disabling the migrations for the 2nd app as well

MIGRATION_MODULES = {
     'app1': None,
     'app2': None
}

Django data migration fails when running manage.py test, but not when running manage.py migrate

3 Answers3