django 1.5 select_for_update considered fragile design

Question

If you were relying on “automatic transactions” to provide locking between select_for_update() and a subsequent write operation — an extremely fragile design, but nonetheless possible — you must wrap the relevant code in atomic(). Since Django 1.6.3, executing a query with select_for_update() in autocommit mode will raise a TransactionManagementError.

Why is this considered fragile? I would have thought that this would result in proper transactionality.

score 10 · Answer 1 · edited May 19 '14 at 19:38

10

select_for_update isn't fragile.

I wrote that "if you were relying on "automatic transactions"" then you need to review your code when you upgrade from 1.5 from 1.6.

If you weren't relying on "automatic transaction", and even more if the concept doesn't ring a bell, then you don't need to do anything.

As pointed out in yuvi's answer (which is very good, thank you!) Django will raise an exception when it encounters invalid code. There's no need to think about this until you see a TransactionManagementError raised by select_for_update.

edited May 19 '14 at 19:38

Maxime Lorant

34,607
19
87
97

answered May 19 '14 at 18:58

Aymeric Augustin

101
4

2

First post of a core developer of the Django framework, *fiesta*! Just there to say thank you for the contribution. Hope there will be other posts from you in the future :-) – Maxime Lorant May 19 '14 at 19:44

yuvi · Answer 2 · 2014-05-11T10:46:34.460

The answer is just around the corner, in the docs for select_for_update (emphasis mine):

Evaluating a queryset with select_for_update in autocommit mode is an error because the rows are then not locked. If allowed, this would facilitate data corruption, and could easily be caused by calling, outside of any transaction, code that expects to be run in one.

In other words, there's a contradicting behaviour between autocommit and select_for_update, which can cause data corruption. Here's the django developer's discussion where they first proposed solving this issue, to quote (again, emphasis mine):

[...] under Oracle, in autocommit mode, the automatic commit happens immediately after the command is executed -- and so, trying to fetch the results fails for being done in a separate transaction.

However, with any backend, select-for-update in autocommit mode makes very little sense. Even if it doesn't break (as it does on Oracle), it doesn't really lock anything. So, IMO, executing a query that is a select-for-update in autocommit mode is probably en error, and one that is likely to cause data- corruption bugs.

So I'm suggesting we change the behavior of select-for-update queries, to error out [...] This is a backwards-incompatible change [...] These projects should probably be thankful -- they were running with a subtle bug that is now exposed -- but still.

So it was an Oracle-only bug, which shown light over a deeper problem that's relevant for all backends, and so they made the decision to make this an error in django.

Atomic, on the other hand, only commits things to the database after it has verifying that there are no errors, thus solving the issue.

What you've written makes sense, but the way the documentation is written suggests that the design was fragile in 1.5. "If you *were* relying on "automatic transactions"".... this to me is saying that using select_for_update in 1.5 to lock rows for update is a fragile design, even with database autocommit turned off (but still using the Django 'autocommit' application implementation) — Trent, May 12 '14 at 09:20
Yes, you're right, it's only fragile when using autocommit, and it's considered as such because it **doesn't actually lock anything**. It's not fragile in a general sense. And notice - the error addresses this issue, since it's only activated on the specific case of `select_for_update` usage *with* autocommit, so it all makes sense — yuvi, May 12 '14 at 14:41
So is the 'extremely fragile design' comment only applicable to Django 1.6+? The way the documentation is written implies that for Django < 1.6, select_for_update is considered fragile — Trent, May 14 '14 at 01:08
@Taras `select_for_update` isn't fragile on its own, only when used together with auotcommit. I'm not sure what are the implications for using it with the Django 'autocommit' app implementation. And technically, since the flawed design errors out from 1.6, then yeah, it's mostly an issue with earlier releases — yuvi, May 14 '14 at 05:27

score 3 · Accepted Answer · edited May 27 '14 at 17:49

Aymeric clarified over email that such a design is fragile because it relies on the implicit transaction boundaries formed by Django 1.5's implicit transactions.

select_for_update(...)
more_code()
save()

This code works in straightforward cases, but if more_code() results in a write operation to the database, then the transaction would close, producing unintended behavior.

Forcing the user to specify the transaction boundaries also leads to clearer code.

django 1.5 select_for_update considered fragile design

3 Answers3