3

To be able to do some decent unittests I want to create a testdatabase. After reading the Django docs, I came to the conclusion that the best way to do this is by creating a testdatabase from the actual database, with the use of fixtures.

To create such a fixture, one is to run:

./manage.py dumpdata appname --indent 2

After dumping, I want to edit the file, and select the data I actually want to use. However, dumping it almost crashes my PC (the scripts starts trashing I think). Also, it seems to first get all data in memory, and then print it all out at once.

The database I'm using is MySQL

Is there a way to get the data out the database without crashing my PC, in a format usable as fixture?

(an alternative which I'm considering is redefining the default datamanager for the objects with lots of rows, to return only needed rows, but that seems like a rather nasty hack, which I'd rather not apply)

markijbema
  • 3,985
  • 20
  • 32
  • You know it's possible to select specific tables inside the app using this syntax? `./manage.py dumpdata appname.modelname --indent 2` – Exelian Mar 01 '11 at 15:46
  • but the problem is one of the tables is really large (it's a zipcode table of all of the Netherlands) – markijbema Mar 01 '11 at 15:58
  • There's no way you're gonna be able to dump more then 13 million records using plain dumpdata. Your best best would indeed be using a custom model manager. Note that this isn't a nasty hack and even the preferred way of handling this in django. – Exelian Mar 01 '11 at 16:08
  • A *custom* handler is in no way a hack, I agree, but to be able to use dumpdata, you need to edit the *default* manager :( – markijbema Mar 01 '11 at 16:10
  • 1
    Hmm, you're right. Well, in that case your better of creating your own dumpdata command (check django/core/manage/commands/dumpdata.py). I must agree that editing (or overwriting) the default manager is a bad idea. – Exelian Mar 01 '11 at 16:15

1 Answers1

2

You should take a look at this ticket and apply the supplied patch. @ramiro recently ran tests to compare the trunk against his latest patch and you can see there is a huge difference in memory consumption at the expense of processing time:

Unpatched runk enter image description here

Filip Dupanović
  • 32,650
  • 13
  • 84
  • 114