4

My django app saves django models to a remote database. Sometimes the saves are bursty. In order to free the main thread (*thread_A*) of the application from the time toll of saving multiple objects to the database, I thought of transferring the model objects to a separate thread (*thread_B*) using collections.deque and have *thread_B* save them sequentially.

Yet I'm unsure regarding this scheme. save() returns the id of the new database entry, so it "ends" only after the database responds, which is at the end of the transaction.

Does django.db.models.Model.save() really block GIL-wise and release other python threads during the transaction?

Charles
  • 50,943
  • 13
  • 104
  • 142
Jonathan Livni
  • 101,334
  • 104
  • 266
  • 359

3 Answers3

7

Django's save() does nothing special to the GIL. In fact, there is hardly anything you can do with the GIL in Python code -- when it is executed, the thread must hold the GIL.

There are only two ways the GIL could get released in save():

  • Python decides to switch threads (after sys.getcheckinterval() instructions)
  • Django calls a database interface routine that is implemented to release the GIL

The second point could be what you are looking for -- a SQL COMMITis executed and during that execution, the SQL backend releases the GIL. However, this depends on the SQL interface, and I'm not sure if the popular ones actually release the GIL*.

Moreover, save() does a lot more than just running a few UPDATE/INSERT statements and a COMMIT; it does a lot of bookkeeping in Python, where it has to hold the GIL. In summary, I'm not sure that you will gain anything from moving save() to a different thread.

UPDATE: From looking at the sources, I learned that both the sqlite module and psycopg do release the GIL when they are calling database routines, and I guess that other interfaces do the same.

Ferdinand Beyer
  • 64,979
  • 15
  • 154
  • 145
  • Don't I/O calls release the GIL anyway? Does it really need to be explicit? – ubik May 17 '12 at 08:03
  • 1
    Yes, it does. Routines written in C don't release the GIL automatically, but the author explicitly has to do it. See http://docs.python.org/c-api/init.html?highlight=gil#releasing-the-gil-from-extension-code – Ferdinand Beyer May 17 '12 at 08:05
  • Ah OK, I was assuming the SQL interface was written in Python. – ubik May 17 '12 at 08:12
  • Thanks for posting the right answer. I want to add one more thing which caused problem in my scenario [Django + mysql], db.close() db.autoCommit(True) – Anil Chahal Jun 30 '15 at 06:35
3

Generally you should never have to worry about threads in a Django application. If you're serving your application with Apache, gunicorn or nearly any other server other than the development server, the server will spawn multiple processes and evade the GIL entirely. The exception is if you're using gunicorn with gevent, in which case there will be multiple processes but also microthreads inside those processes -- in that case concurrency helps a bit, but you don't have to manage the threads yourself to take advantage of that. The only case where you need to worry about the GIL is if you're trying to spawn multiple threads to handle a single request, which is not usually a good idea.

The Django save() method does not release the GIL itself, but the database backend will (in most cases the bulk of the time spent in save() will be doing database I/O). However, it's almost impossible to properly take advantage of this in a well-designed web application. Responses from your view should be fast even when done synchronously -- if they are doing too much work to be fast, then use a delayed job with Celery or another taskmaster to finish up the extra work. If you try to thread in your view, you'll have to finish up that thread before sending a response to the client, which in most cases won't help anything and will just add extra overhead.

Andrew Gorcester
  • 19,595
  • 7
  • 57
  • 73
  • My system is not a web application. I'm using django purely for the ORM, no web server involved – Jonathan Livni May 18 '12 at 06:58
  • Hmm, I see. Well, if you're doing a lot of processing in python (CPU time outside of DB access) then you could use the multiprocessing library or just launch multiple processes, which is what I do with my scripts that use Django for the ORM (mine take days to run even with multiple processes). But if you do use threads they can at least multiplex on database i/o operations because the actual blocking i/o will release the GIL. – Andrew Gorcester May 18 '12 at 14:40
-4

I think python dont lock anything by itself, but database does.

Denis
  • 7,127
  • 8
  • 37
  • 58
  • the database is remote thus over an IP connection, therefore I'm not sure what you mean, it can't block over the connection, the question is what happens locally – Jonathan Livni May 17 '12 at 07:18
  • Yes I know what is GIL, but Django dont use threads, and I think it dont use GIL. – Denis May 17 '12 at 07:20
  • django uses a database backend. So is this actually a backend specific question? – Jonathan Livni May 17 '12 at 07:21
  • Django working with psycopg2 as Postgres backend and psycopg cant into threads like and database, when you start wtite to table its lock and there are no mean use threads. – Denis May 17 '12 at 07:24
  • Denis, could you rewrite that, I don't understand the English, and as this is my specific case I'm interested to know about it – Jonathan Livni May 17 '12 at 07:25
  • Sorry but my English is poor and sometimes I can't properly express my thoughts. But I want to tell what problem not in python's GIL it is in database, SQL databases can not be in the multi-threading. – Denis May 17 '12 at 07:29