How to deal with a `relation "cms_disclaimerpanel" already exists` and ProgrammingError: column "http_request_lang" of relation "xyz" does not exist

Question

I have a rather annoying issue when trying to send my merge to my automated tests on circle CI.

Just for the context, I've inherited a project where the authors are no longer working at my current work.

I'm working on django and I've done a merge, from my local dev branch to my local master branch. The merge went well. However, when starting the django server through a manage.py runserver, it gives me the warning Your project may not work properly until you apply the migrations for app(s)[...].

When doing the manage.py migrate, I'm running into the first issue:

1- django.db.utils.ProgrammingError: relation "cms_disclaimerpanel" already exists

I fix the issue by manually editing the migration file, commenting the following lines

 #       migrations.CreateModel(
 #           name='DisclaimerPanel',
 #           fields=[
 #               ('abstractpanel_ptr', models.OneToOneField(auto_created=True, on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='cms.AbstractPanel')),
 #               ('title', models.CharField(blank=True, max_length=1024, verbose_name='title')),
 #               ('show_title', models.BooleanField(default=True, verbose_name='show title')),
 #               ('subtitle', models.TextField(blank=True, verbose_name='content')),
 #               ('show_subtitle', models.BooleanField(default=True, verbose_name='show subtitle')),
 #               ('alignment', models.CharField(choices=[('left', 'left'), ('center', 'center')], default='center', max_length=10, verbose_name='text alignment')),
 #               ('button', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, to='cms.Link')),
 #           ],
 #           options={
 #               'verbose_name': 'Disclaimer Panel',
 #           },
 #           bases=('cms.abstractpanel',),
 #       )

Then the second issue happened, while carrying my manage.py migrate

2 - ProgrammingError: column "http_request_lang" of relation "cms_dynamicsettings" does not exist

I fix the issue by manually editing the migration file, commenting the following lines

#operations = [
#    migrations.RemoveField(
#        model_name='dynamicsettings',
#        name='http_request_lang',
#    ),
#]

The manage.py was able to run entirely. Then I ran manage.py makemigrations and it gives me this last file

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

    dependencies = [
        ('cms', '0088_merge_20190411_1655'),
    ]

    operations = [
        migrations.CreateModel(
            name='DisclaimerPanel',
            fields=[
                ('abstractpanel_ptr', models.OneToOneField(auto_created=True, on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='cms.AbstractPanel')),
                ('title', models.CharField(blank=True, max_length=1024, verbose_name='title')),
                ('show_title', models.BooleanField(default=True, verbose_name='show title')),
                ('subtitle', models.TextField(blank=True, verbose_name='content')),
                ('show_subtitle', models.BooleanField(default=True, verbose_name='show subtitle')),
                ('alignment', models.CharField(choices=[('left', 'left'), ('center', 'center')], default='center', max_length=10, verbose_name='text alignment')),
                ('button', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, to='cms.Link')),
            ],
            options={
                'verbose_name': 'Disclaimer Panel',
            },
            bases=('cms.abstractpanel',),
        ),
        migrations.RemoveField(
            model_name='dynamicsettings',
            name='http_request_lang',
        ),
    ]

With the changes above, I can runmanage.py runserver.

I'm then adding these 3 files to my merged branch and create a distant repository for my branch.

Each new repository created is run through the unit test and there lies the issue as it does not take into account my three newly commited files.

It gives me the following error, the same error as in the point 2 (see above).

ERROR:  relation "cms_dynamicsettings" does not exist at character 1508

Using my dev environnement as a template, my guess would be that circle ci is replicating the same issue I've encountered and that I manually fixed.

The questions are the following:

Is there a way to drop my model cms_disclaimer before it runs through that damn migration file? If yes, how?
Is there a way to not take into account the migration file and tell it not to drop the column http_request_lang
My last question is why the manage.py makemigrations does not see the changes in the database?

One more information:

The database was built with the branch master. I checkout on another branch based on master , merged my dev branch into master and then do manage.py migrate.

Any info will be more than welcome as I'm loosing my sanity.

Thanks.

`makemigrations` does not have anything to do with the database state. It's sole purpose is to ensure that the result of all the migrations files corresponds to the state of your models. Then `migrate` ensures all migrations are applied in the database. It also tracks which migrations were applied in a special table of the database. You should probably not change the migration files manually without knowing exactly why the first migration didn't work. Since this is local, you should try to completely clear your database and migrate from scratch (this is also what your unit tests are doing). — dirkgroten, May 24 '19 at 13:22
@dirkgroten this is what I understood. But in QA, there is already a database in place. I'm not going to ask QA to drop the database, if you see where I'm coming to — Andy K, May 24 '19 at 13:28
indeed @dirkgroten migrate does not work on QA and this is why I have the issue. — Andy K, May 24 '19 at 13:29
the table you want to look at is "django_migrations" (inside your db). Check that. It should tell you which migration files were already applied in the existing db. Delete all migration files that were added after the last applied migration. And run `make migrations` again after that. — dirkgroten, May 24 '19 at 13:33
@dirkgroten I'm able to see all the files applied. What do you mean by `Delete all migration files that were added after the last applied migration`? — Andy K, May 24 '19 at 13:42
the last file in my db for that app is `0080_disclaimerpanel`. Should I delete all the migration files for that app , after the number 80? @dirkgroten — Andy K, May 24 '19 at 13:43
yes. If in your code repo all migration files are numbered sequentially (and there's only one app) then delete > 0080. But if there are more apps, you'll need to figure out the last applied number for each app. Strictly speaking, the numbers are not relevant (django uses the `dependencies` variable to figure out the order) but hopefully your previous dev team kept it to sequential. — dirkgroten, May 24 '19 at 13:46
@dirkgroten I deleted the file 80 , to no avail. It is not doing the work... -_- — Andy K, May 24 '19 at 13:59
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/193891/discussion-between-dirkgroten-and-andy-k). — dirkgroten, May 24 '19 at 14:06
@dirkgroten issue is there are models that have been created after 0080 — Andy K, May 24 '19 at 14:06

score 1 · Accepted Answer · answered May 24 '19 at 14:52

The procedure in such a situation is to ensure you first get to a state that is in sync with your database (assuming you can't drop your database because the system is in production).

Check in your production database the table "django_migrations" and look at the last migration that was applied for each of your apps.
In your code, delete all migration files that were added after that last migration. Do that for each app. Make sure all the migration files applied to your database are present in your code repo.
Run manage.py migrate, which should do nothing ("nothing to migrate") if your migration files are in sync with your db.
Run manage.py makemigrations which will create one extra migration file that reflects all the changes in your models with respect to your database.
Run manage.py migrate and everything should be working now.

Some things to worry about when doing this:

Make sure all the databases you are using (in all environments) are in sync with the production database.
Make sure no one is working on a different branch where some other migrations might exist.

How to deal with a `relation "cms_disclaimerpanel" already exists` and ProgrammingError: column "http_request_lang" of relation "xyz" does not exist

1 Answers1