Fine tuning Django Apache mod_wsgi

Question

First off I am looking for some help setting up my Django and Apache settings to fine tune it for my site for the best performance cost effectiveness.

Server info: Django 1.3 Webfacion server, 80mb ram allotted currently. I haven't really touched the http.conf that Webfaction already provides. My site hasn't been released yet but it is running in production and taking up about 17mb with the main process and about 30mb overall. I did some load testing today and it never went over 20mb for the main process.

Site info: Single Django app, the performance is great, everything loads fast (very low traffic now). I only use a few templating files and they are being served my nginx. I have only a few models and all of the handlers pull data with paging and none of them load more than 25 records in a QuerySet at one time. The templating is very simple, just puts some media in the html and a few json variables, the client side ui is all driven by js. I am currently not using the django caching system. I was planning on upgrading and getting 80mb more of ram, the last thing I want to do is for the site to become popular and crash.

Are there some tools out there to simulate traffic? What would 80mb of more ram really help? Is there a place where I can get professional advice by people that really know this stuff? (I would be willing to pay them).

Any answers would be great. Thanks in advance.

Update: everything runs fine I just want to fine tune it. here is the httpd.conf file:

ServerRoot "/home/pllee/webapps/django/apache2"

LoadModule dir_module        modules/mod_dir.so
LoadModule env_module        modules/mod_env.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module       modules/mod_mime.so
LoadModule rewrite_module    modules/mod_rewrite.so
LoadModule setenvif_module   modules/mod_setenvif.so
LoadModule wsgi_module       modules/mod_wsgi.so

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog /home/pllee/logs/user/access_django.log combined
ErrorLog /home/pllee/logs/user/error_django.log
KeepAlive Off
Listen 22504
MaxSpareThreads 3
MinSpareThreads 1
ServerLimit 1
SetEnvIf X-Forwarded-SSL on HTTPS=1
ThreadsPerChild 5
WSGIDaemonProcess django processes=5 python-path=/home/pllee/webapps/django:/home/pllee/webapps/django/lib/python2.7 threads=1
WSGIPythonPath /home/pllee/webapps/django:/home/pllee/webapps/django/lib/python2.7
WSGIScriptAlias / /home/pllee/webapps/django/SiteName.wsgi

Answer 1

You can comment out the line:

WSGIDaemonProcess django processes=5 \
  python-path=/home/pllee/webapps/django:/home/pllee/webapps/django/lib/python2.7 \
  threads=1

for a start. This line is doing absolutely nothing except for causing five processes to be started which then sit there idle and do nothing.

This is the mistake WebFaction has had for a while. They specify a daemon process group but then don't actually define a WSGIProcessGroup directive to cause mod_wsgi to run your WSGI application in it.

Next thing you can do since you are only running a single WSGI application is add:

WSGIApplicationGroup %{GLOBAL}

This will cause WSGI application to run in main interpreter of the processes and avoid creation of a second sub interpreter. Use of the main interpreter will avoid various problems with third party C extension modules that aren't implemented to run properly in sub interpreters.

Those are simple things you can do although they are just grabbing back some memory.

Overall, in practice the bottlenecks are not going to be in the underlying web hosting mechanism. As such, you might want to be looking at what monitoring tools you can use to monitor things and identify bottlenecks. For development environment you have Django debug toolbar but it is going to be quite limiting and only useful to debug a known issue and not really identify the source of the problems. For the latter you really need production monitoring tools in which case you may want to have a look at New Relic which for Python is in BETA at the moment. Yes I work at New Relic and it is want I am doing, but even though it is a paid service for full features, it does have a Lite version for free. During the BETA though it is free for the full features so may be worth a look as it may help you get your application running better so you aren't as worried about an increase in traffic later.