MemoryError while loading huge initial data

Question

I have the initial data from my old database which takes around 6GB. I could "dumpdata" my old database without any problem. But when I attempted to restore them to the new database, I got the MemoryError:

    python manage.py loaddata fixtures/initial_data.json
    MemoryError: Problem installing fixture 'fixtures/initial_data.json':

Is there any way to make loaddata work with chunks or is it possible to load that big file?

How many tables does the fixture contain? If more than one, you could split one table per file. — augustomen, Aug 31 '13 at 20:46

Kostyantyn · Answer 1 · 2014-05-17T07:23:40.523

8

I've wrote this script, which is a fork of django's dumpdata, but dumps data in chunks to avoid MemoryError. And then load these chunks one by one.

Script is available at https://github.com/fastinetserver/django-dumpdata-chunks

Example usage:

1) Dump data into many files:

mkdir some-folder

./manage.py dumpdata_chunks your-app-name
--output-folder=./some-folder --max-records-per-chunk=100000

2) Load data from the folder:

find ./some-folder | egrep -o "([0-9]+_[0-9]+)" | xargs ./manage.py loaddata

PS. I used it to move data from Postgresql to MySQL.

edited May 17 '14 at 07:23

answered Jan 31 '14 at 06:04

Kostyantyn

5,041
3
34
30

2

Why would you move from Postgresql to MySQL? – Cesar Mar 21 '16 at 05:47
As always: client's request :) Reasons: client want to integrate data with some php framework that supports only mysql. – Kostyantyn Mar 21 '16 at 08:05

score 1 · Accepted Answer · answered Aug 31 '13 at 20:55

1

For large database use backup tools for dumping database data instead "django dumpdata". To load database data use restore tools instead "django loaddata".

answered Aug 31 '13 at 20:55

P̲̳x͓L̳

3,615
3
29
37

I think that's the only one way. The problem is that one database is Sql Server and the other is Postgres, it will be hard to copy foreign keys. – cem Aug 31 '13 at 21:00
@user2211623: Does the standard technique of copying the data without FK constraints, and then re-imposing the constraints at the end, not work? – Mechanical snail Aug 31 '13 at 21:05
@user2211623: Article [Microsoft SQL Server to PostgreSQL Migration by Ian Harding](http://wiki.postgresql.org/wiki/Microsoft_SQL_Server_to_PostgreSQL_Migration_by_Ian_Harding) – P̲̳x͓L̳ Aug 31 '13 at 21:16
Thank you. I think I will do it in that way. I also couldn't find anything about this MemoryError anywhere. So, that. – cem Aug 31 '13 at 21:21

MemoryError while loading huge initial data

2 Answers2