1

I'm utilizing a Apache + mod-wsgi + django setup on Windows with python 2.7. It seems that the only setup that is possible on Windows for mod-wsgi is having 1 process with many child worker threads. Because of this, it seems that some work may be subject to GIL. I've noticed that database requests don't appear to cause locks, but some python processing does cause locking/slowdown.

For example:

If I'm processing a large xml file using lxml via soaplib, it causes massive slowdowns. Reading the documentation, it seems the solution to this is to use WSGIApplicationGroup %{GLOBAL}. Side note, does this even work in Windows?

If I'm doing a large list processing job via python natively that is CPU intensive, it also seems to slow other requests.

I'm wondering if there is a general class of work that will cause django/python to lock until it's finished. And if so, what are some best practices to avoid these issues?

Ringil
  • 6,277
  • 2
  • 23
  • 37

2 Answers2

2

Setting WSGIApplicationGroup %{GLOBAL} although advisable if using lxml, isn't going to solve performance issues. It protects against deadlocks and crashes due to lxml not being incompatible with sub interpreters. Thus completely different issue.

As to performance, if you have many requests performing CPU intensive Python only code, then there will be some GIL contention, but that will only slow the throughput overall and not block concurrent requests which are also doing CPU intensive work. This is because the Python interpreter will cause control by a thread to be implicitly yielded every certain number of Python byte code instructions so that other threads can run.

The bigger problem is where you are using a module which has a C extension component and it is doing CPU intensive long running tasks and what it is doing means it has to operate on Python data structures and so cannot release the GIL to allow other threads to run. In other words, C code which takes a long time and doesn't release the GIL locks out other threads.

If you are seeing this sort of problem, because Windows doesn't allow multi process Apache, you would have to use some sort of backend task queuing system which can farm out the actual work to separate processes somehow. On UNIX systems you would use Celery, or Redis Queue for that. What your options on Windows are I have no idea.

Graham Dumpleton
  • 57,726
  • 6
  • 119
  • 134
  • So, if I'm using Python only code and interpreting what you've said correctly, the following scenario should be what's happening: Let's say we have 2 requests A and B with A coming in 1 second before B. If A takes say 100 seconds to process and B takes 1 second to process, does what you said mean that B be processed within the 100 seconds, but take longer than 1 second to process? – Ringil Jun 11 '15 at 13:24
  • How long it takes will depend on how heavily each is using the CPU. If both tasks require 100% CPU for the whole time, then theoretically in the worst case, request B would take about 2 seconds and A 101 seconds. This is because during the overlap, each would only be able to get at most 50% CPU and so would be slowed by half during the overlap. In practice it is not going to be that clear cut though and it really depends on what Python modules are used in your code and what they are doing for you. – Graham Dumpleton Jun 11 '15 at 20:46
0

As why you get "massive slowdowns" when doing some CPU or memory intensive computations, it doesn't necessarily has to do with the GIL, it might just be your system not being able to cope with the load. Scaling heavy processings is usually dealt with using a multiple servers setup (eventually with parallelisation).

bruno desthuilliers
  • 75,974
  • 6
  • 88
  • 118