Apache django mysql very slow

Question

I am using Django, wsgi with apache, but the server is really slow when I try to insert data into mysql server. It takes 1 second to insert 200 rows in a table. The strange thing is that only 3% of CPU and memory is used by apache and mysql. I am sure there is some setting wrong, but I cannot find it.

Here is my apache config file, wsgi setting and 'top' command result.

I really appreciate if someone could help me a little bit. Thanks!

My apache config file:

WSGIScriptAlias / "/home/jiechao/EZ_Green/apache/ems_django.wsgi"
ServerName www.ezgreenbuildings.com
WSGIDaemonProcess ezgreenbuildings.com processes=2 maximum-requests=1000 threads=25
WSGIProcessGroup ezgreenbuildings.com

wsgi file:

os.environ['DJANGO_SETTINGS_MODULE'] = 'ems.settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

sys.path.append("/home/jiechao/EZ_Green/django")
sys.path.append("/home/jiechao/EZ_Green/django/ems")

top command result:

top - 15:56:02 up 25 days, 19:30,  1 user,  load average: 0.74, 0.24, 0.11
Tasks: 112 total,   1 running, 111 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  0.7%sy,  0.0%ni, 58.5%id, 40.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4124236k total,  3242020k used,   882216k free,   157960k buffers
Swap:  4192252k total,      440k used,  4191812k free,  2750284k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6487 www-data  20   0  101m  63m 7288 S    1  1.6   0:03.02 apache2
30558 mysql     20   0  313m  44m 7020 S    1  1.1   0:54.98 mysqld
  258 root      20   0     0    0    0 D    1  0.0   1:09.21 jbd2/sda1-8
10243 root      20   0     0    0    0 S    0  0.0   0:00.04 kworker/1:1
14410 jiechao   20   0  208m 8268 3296 S    0  0.2   4:39.92 python
    1 root      20   0  3504 1876 1276 S    0  0.0   0:01.58 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.13 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:45.39 ksoftirqd/0
    5 root      20   0     0    0    0 S    0  0.0   0:00.40 kworker/u:0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S    0  0.0   0:03.85 watchdog/0

After Edited.

I added 'WSGIProcessGroup' now, so it should be invoking daemon mode now. I am using Prefork for MPM, but I don't understand how to see the settings of MPM.

For site visits, I think 1000 visits per second is what I would expect.

Is that 1000 inserts from a single connection? Or 1000 connections, each inserting one row? And, are you talking about a single table? MyISAM or InnoDB? What is the setting of `innodb_flush_log_at_trx_commit`? — Rick James, Dec 24 '19 at 07:23

Graham Dumpleton · Accepted Answer · 2013-01-01T21:30:26.807

You have a few problems.

You are missing WSGIProcessGroup directive, meaning that WSGIDaemonProcess is redundant and not even being used. You are there running in embedded mode and not daemon mode. Embedded mode is not a good idea unless you know how to set up Apache properly. Read:

http://blog.dscpl.com.au/2012/10/why-are-you-using-embedded-mode-of.html

If you had managed to configure it so daemon mode was used properly, your processes/threads settings are insanely high and would likely just chew up lots of memory on your system, potentially causing it to start swapping and slow your system down.

Based on missing WSGIProcessGroup, you are at least not invoking that yet.

How big a problem you have created for yourself by running in embedded mode really depends on what Apache MPM you are using and what the MPM settings are.

So, before can debug further, you need to work out whether you are using Apache prefork or worker MPM and also what MPM settings you are using. Get that information and edit your question, adding the additional information.

Also give a realistic estimate of how many requests/sec your site is actually handling. That way can evaluate how wrong the MPM and daemon process process/threads settings are.

For additional reading, see:

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation

What is in there will help you work out which MPM you are using and whether embedded mode or daemon mode is being used. Plus whether mod_python is also being loaded which can cause another world of pain as well.

UPDATE 1

The Apache MPM settings are in the Apache configuration files. Go read the Apache documentation about the MPM settings so you know what they are. Especially with prefork MPM, if you don't understand what they are and how to set them up properly to tune your site, you will fail.

As for the claim of 1000 visits per second, if that is true then you are going to slow things down immensely by using maximum-requests of 1000 as your web application processes will be restarting every couple of seconds and the delays in doing that would be slowing everything down. If one can avoid it, you should never use maximum-requests on a production system.

Frankly though I am dubious about claim that you are handling 1000 visits per second. A Django application using a database is highly unlikely to be able to handle anywhere near that on one server. Is that actual traffic or is is just your dreams of what you want to be able to handle? What actual performance monitoring are you running to support the claim you are handling that number of requests per second?

Thanks for your patient answer. 1000 visits per second is what I would expect, not the actual traffic now. And maybe in a few years. If you don't think it's realistic, could you give me some suggestions on how to achieve that goal? — Jiechao Li, Jan 02 '13 at 06:22
Beyond server setup issues, you really need to get some monitoring in place, otherwise you are going to be blind as to where the real bottlenecks are. Go watch http://lanyrd.com/2012/pycon/spcdg/ — Graham Dumpleton, Jan 02 '13 at 09:55

Apache django mysql very slow

1 Answers1