How to customize Django loaddata so can perform logic before loading the model data

Question

I backed up DB with manage.py dumpdata, I now have a file I wish to use with manage.py loaddata, but I wish to perform some customised logic to the data before loading it. This is because I am migrating the data to a different structure. How does one get the data into python objects so I can perform my customised logic?

Answer 1

The following code gives one 2 ways of loading data from fixture file:

1. Using Python's native JSON module. The advantage of this is event if your models have changed, you can load the data, manipulate it into the required form and save it.
2. Using Django deserializer. The advantage of this is that it create the model instance for you, but if you are migrating code, and the model instance code does not exist any more, this could be an issue.

""" Refer to help variable below
"""
import json

from pathlib import Path

from django.core import serializers
from django.core.management.base import BaseCommand

class Command(BaseCommand):
    help = "Customized load data for DB migration"

    def iter_json(self, file):
        """ Iterates dumpdata dson file and yields a model name / field dict pairs.

        e.g.
            model = 'common.sitesettings'
            fields = {'name': 'ACME Products', 'timezone': 'Africa/Johannesburg'}
        """
        with open(file, "r") as f:
            data = json.loads(f.read())
        for obj in data:
            yield obj['model'], obj['fields']

    def iter_data(self, file):
        """ Iterates dumpdata dson file and yields model instances.

        The downside to this approach is that one requires the code for the models,
        which is not neccessarily the case when migrating data/
        e.g.
            model_instance = <SiteSettings: ACNE Products ()>
        """
        data  = self.parse_json(file)
        with open(file, "r") as f:
            data_str = f.read()
        for item in serializers.deserialize("json", data_str):
            # item.save() would save it (unconfirmed)
            model_instance = item.object
            yield model_instance

    def add_arguments(self, parser):
        parser.add_argument(
            'file',
            type=str,
            help='Location of the manage.py dumpdata backup file',
        )

    def handle(self, **options):
        file = options['file']
        if not Path(file).is_file():
            print(f"File '${file}' does not exist. EXIT.")
            return

        print(f"START - Customised loaddata")
        for model, fields in self.iter_json(file):
            print(model, fields)
        for instance in self.iter_data(file):
            print(instance)
        print(f"COMPLETE.")