14

In the latest Django (2.2), when I add a new field to a model like this:

new_field= models.BooleanField(default=False)

Django runs the following commands for MySQL:

ALTER TABLE `app_mymodel` ADD COLUMN `new_field` bool DEFAULT b'0' NOT NULL;
ALTER TABLE `app_mymodel` ALTER COLUMN `new_field` DROP DEFAULT;
COMMIT;

While this works when everything is updated, this is very problematic because old versions of the application can no longer create models after this migration is run (they do not know about new_field). Why not just keep the DEFAULTconstraint?

Petter
  • 37,121
  • 7
  • 47
  • 62
  • Is this purely a theoretical question, or are you also looking for a solution? – Lord Elrond Sep 30 '19 at 01:23
  • 1
    I am looking for a way to safely add a new field without `null=True`. Adding a `BooleanField` with `default=False` does not work. That will make old versions of the app crash. – Petter Sep 30 '19 at 08:53
  • I don't understand this last comment. You either want a field that has two states or three states: `models.BooleanField()` or `models.BooleanField(null=True)`. Either way, will have a default value of `None` unless you specify a default, as you have in you example. – gregory Oct 02 '19 at 04:34

2 Answers2

15

Why not just keep the DEFAULT constraint?

Because Django handles the default model field option at the application level, not the database level. So the real question is why it sets the DEFAULT constraint at all.

On the first point, historically Django has not supported database-level defaults for model fields. There has always been some interest in changing that (the first issue on the subject is 18 years old), and certainly other frameworks (Rails, I think, and SQLAlchemy) have shown that it is possible.

And recently Django has added that option with a separate field option called db_default. The behavior of default will remain the same, though, and there are good reasons beyond backwards compatibility for handling defaults at the application level. Such as: the ability to express arbitrarily complex computations; not having to worry about subtle incompatibilities across database engines; the ability to instantiate a new instance in code and have immediate access to the default value; the ability to present the default value to users in forms; and more.

Now, adding a new non-nullable field to an existing database is a very different use case. In that situation, you have to provide a default to the database for it to perform the operation. makemigrations will try to infer the right value from your default option if it can, and if not it will force you to specify a value from the command line. So the DEFAULT modifier is used for this limited purpose and then removed.

As you've noticed, the lack of database-level defaults in Django can make continuous deployment harder. But the solution is fairly straightforward: just re-add the default yourself in a migration. One of the great benefits of the migrations system is that it makes it easy to make arbitrary, repeatable, testable changes to your database outside of Django's ORM. So just add a new RunSQL migration operation:

operations = [
    # Add SQL for both forward and reverse operations
    migrations.RunSQL("ALTER TABLE app_mymodel ALTER COLUMN new_field SET DEFAULT 0;",
                      "ALTER TABLE app_mymodel ALTER COLUMN new_field DROP DEFAULT;")
]

You can put that in a new migration file or simply edit the automatically generated one. Depending on your database and its support for transactional DDL, the sequence of operations may or may not be atomic.

Kevin Christopher Henry
  • 46,175
  • 7
  • 116
  • 102
  • 1
    I think this is the correct answer. I am not a fan of RunSQL-migrations myself, but perhaps I should reconsider. – Petter Oct 03 '19 at 11:12
  • FYI: Django [recently implemented](https://github.com/django/django/commit/7414704e88d73dafbcfbb85f9bc54cb6111439d3) database defaults in the development version of Django with the [db_default](https://docs.djangoproject.com/en/dev/ref/models/fields/#db-default) keyword. – sytech Aug 30 '23 at 00:13
5

I found this ticket from 2 years ago: https://code.djangoproject.com/ticket/28000
It is stated in there that:

Django uses database defaults to set values on existing rows in a table. It doesn't leave the default values in the database so dropping the default is correct behavior. There might be an optimization not to set/drop the default in this case -- I'm not sure it's needed since the column isn't null. A separate ticket could be opened for this.

I also saw the same reference in another question here: Django Postgresql dropping column defaults at migrate

And searching a bit more I came upon this SO question: Django implementation of default value in database that led to the code of the _alter_field method from django.db.backends.base.schema where this comment exists:

# When changing a column NULL constraint to NOT NULL with a given
# default value, we need to perform 4 steps:
#  1. Add a default for new incoming writes
#  2. Update existing NULL rows with new default
#  3. Replace NULL constraint with NOT NULL
#  4. Drop the default again.

Although the last one is about altering an existing nullable field to a non-nullable, this seems to be the way that Django handles the default case :/

John Moutafis
  • 22,254
  • 11
  • 68
  • 112
  • 1
    Thanks for the references! I am not sure I understand *why* though. And I still don't understand how I can ever add a new field safely without `null=True`, but that's a separate question. – Petter Sep 30 '19 at 08:51
  • @Petter Maybe if you try to add the new field and then when you are about to apply the migration `./manage.py migrate` it will ask you for a default value. (I haven't tried this so I am just suggesting to give it a try) – John Moutafis Sep 30 '19 at 09:42
  • 1
    That question should appear when creating the migration, not applying. – Petter Sep 30 '19 at 10:48
  • @Petter True that. So does this work or results in the same behavior? – John Moutafis Sep 30 '19 at 10:52
  • 1
    You seem to be asking about adding the field without a default value and null=False? That will also crash with old versions of the software afterwards. – Petter Sep 30 '19 at 10:57
  • @Petter Yes, I was asking about that because I have never used it in the past. From what I have seen around looking for a possible solution, sadly only the `null=True` seems to be the way to go :/ – John Moutafis Sep 30 '19 at 19:02
  • How is this answering the question? OP is talking about *adding* a new field -- not migrating an old field and its values to something new. – gregory Oct 02 '19 at 04:11
  • @gregory This answer addresses the reasons why he cannot do it the way he is trying to and leads to the conclusion that probably the only way is to declare any new fields as `null=True`. The part that speaks about migration (which I specifically state) is just another sample of the behavior that Django has regarding the `default` field. – John Moutafis Oct 02 '19 at 06:43