Django: How to migrate from ManyToMany to ForeignKey?

Question

I'm building a REST API and server using Django and Django Rest Framework. We are using a postgres database.

I need to simplify a badly designed relation. We have a model (House) that has a ManyToMany relationship to another (City). In reality it would be enough when this is a ForeignKey relationship.

I googled and couldn't find any blog posts or docs how to properly migrate in this direction. I could only find the other way (FK to M2M).

I'm 98% sure that all the data on the server will be consistent with a FK relationship (meaning I'm pretty sure all houses only have one city). We need to change the relationship for several reasons and aren't able to keep the M2M.

I'm afraid to just change the model and running makemigrations and migrate. I was wondering, how do you properly migrate from M2M to FK? Are there any caveats I have to take into account? How can I deal with data if surprisingly there are houses with multiple city's? The data set is still quite small (less than 10k entries) if that matters.

Thank you very much.

Answer 1

EDIT First create a DB backup

First create a new temporary FK relationship

_city = models.ForeignKey(...)

and make a migration

python manage.py makemigrations
python manage.py migrate

You then need to create a new data migration in the relevant app by creating an empty migration file:

python manage.py makemigrations --empty myhouseapp

and in that file manually assign the new relationship from the M2M to the FK. It will look something like:

from django.db import migrations


def save_city_fk(apps, schema):
    City = apps.get_model('myapp', 'City')
    House = apps.get_model('myapp', 'House')
    for house in House.objects.all():
        house._city = house.cities.all().first()  # Or whatever criterea you want
        house.save()


def save_city_m2m(apps, schema):
    # This should do the reverse
    City = apps.get_model('myapp', 'City')
    House = apps.get_model('myapp', 'House')
    for house in House.objects.all():
        if house._city:
            house.cities.add(house._city)


class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        migrations.RunPython(save_city_fk, save_city_m2m)
    ]

Remove the M2M field, and create another migration.

python manage.py makemigrations

Rename the FK from _city to city and create a final migration

python manage.py makemigrations

Then migrate:

python manage.py migrate

Answer 2

Based on Timmy's answer here is what I did:

1. I added a field like this city = models.ForeignKey(City, related_name='has_houses', blank=True, null=True) to avoid the related_name for reverse relations and to have the FK blank. Then I ran makemigrations and migrate.

2. Next, I ran python manage.py makemigrations --empty houses because my app is named houses.

3. I added the code for the migration (see below). Then I ran migrate.

4. I deleted the M2M field and the related_name, null and blank constraints and ran makemigrations and migrate one last time.

Code for the migration:

# -*- coding: utf-8 -*-
# Generated by Django 1.11.15 on 2019-02-15 09:09
from __future__ import unicode_literals

from django.db import migrations


def save_city_fk(apps, schema):
    City = apps.get_model('cities', 'City')
    House = apps.get_model('houses', 'House')
    for house in House.objects.all():
        house.city = house.cities.all().first()
        house.save()


class Migration(migrations.Migration):

    dependencies = [
        ('houses', '0003_auto_20190215_0949'),
    ]

    operations = [
        migrations.RunPython(save_city_fk),
    ]

Answer 3

First obviously you need to make sure (by making the appropriate query) that you really only have one city per house. If there are houses which have more than one city, you need to resolve the conflict by deleting cities from the relationship, splitting houses etc.

After that, you can do it in steps:

• create a new FK in House, migrate and populate it with the one city id from the old m2m relationship
• rename the m2m field, migrate, then give the new FK the name of the old m2m field if desired and migrate again
• check if your queries work and adapt them as necessary
• when satisfied that everything works, delete the old m2m field – only after migrating this step will you lose the ability to roll back the db

Answer 4

for M2M relation it is better to build a new MOdel to represent the relation. following is the example of an`employee having multiple designation and designation obtained by multiple employees

class Designation(models.Model):
    desig_id = models.AutoField(primary_key=True)
    title = models.CharField(max_length=60, unique=True)
    job_discription = models.CharField(max_length=2000)

    def __str__(self):
        return self.title

class Employee(AbstractUser):
    middle_name = models.CharField(max_length=20, blank=True,  null=True)
    basic_salary = models.FloatField(default=1)
    designation = models.ManyToManyField(Designation, default=None, blank=True,
                                     through='EmployeeDesignation')
    certification = models.ManyToManyField(Certification, default=None, null=True, blank=True,
                                        through='EmployeeCertification')
    emp_img = models.FileField(default=None,upload_to='employees')
    leaves_allowed = models.IntegerField(default=25)
    leave_balance = models.IntegerField(default=25)
    leave_count = models.IntegerField(default=0)
    objects = EmployeeManager()

    def __str__(self):
        return self.first_name+' '+self.last_name

class EmployeeDesignation(models.Model):
    desig = models.ForeignKey(Designation, on_delete=models.CASCADE)
    emp = models.ForeignKey(Employee, on_delete=models.CASCADE)

    class Meta:
        unique_together = (('emp', 'desig'),)

Answer 5

I have seen that the answers are saying that you have to create multiple migration files to do it but this is not great practice and after some time you may squash your migrations together. Just create an empty migration or create an automatic migration by using:

python manage.py makemigrations

and provide a default value for the existing instances.

After that, you can change the file and modify the changes and apply the data migration without losing any data. Create the functions and run them after adding the new fields and after that, you can delete them and rename the new field properly.

migrations.AddField(...),

migrations.RunPython(save_city_fk),

migrations.RemoveField(...),

migrations.AlterField(...),

The order is important.