0

I have a django site on webfaction that uses apache + mod_wsgi.

Site is getting around 1000 requests per minute.

But it makes some calculations, so request takes about 5-10 seconds.

I use the following configuration

StartServers         2
MinSpareThreads      10
MaxSpareThreads      25
ThreadLimit          25
ThreadsPerChild      25
MaxClients           75
MaxRequestsPerChild   1000

threads=15 processes=12

The problem is high CPU usage and it takes time to process a simple static page without calculations (looks like Apache queued the request).

So what I want is for Apache to quickly accept requests.

I'm totally lost because of number of parameters, I also don't quite understand what they mean. What do we need StartServers and MaxRequestWorkers for?

Any help and/or explanations will be highly appreciated.

I have 8GB of RAM.

Apache MPM Worker.

mod_wsgi 4.4.21.

Thank you in advance.

Paul R
  • 171
  • 6

1 Answers1

0

StartServers is the number of number of server processes you start with, and MaxRequestWorkers is the number of threads per process. The Webfaction settings should be reasonable in most situations although a thousand requests a minute may need some tuning, but probably mostly in the application.

In normal use httpd would take a request, pass it to mod_wsgi and wait for its return, which should be near instantaneous, so what is actually taking its time is whatever the python script is doing. Your httpd worker threads are therefore in a wait state and will build up as requests come in while other requests are being processed, so even a static page will end up waiting if your threads are occupied.

Look at what the application is doing and for solutions. You may be able to cache queries using memcached or similar. If the time that your application takes to process a request is unavoidable, look at making it asynchronous using a message queue like Celery, so rather than having your web server wait for responses, you can poll for them using browser side scripting.

Splitting the static page serving from the dynamic will also improve the response. If possible you could run multiple sets of workers, or pass static page and object serving to nginx, which is a more common way of handling wsgi.

Another method would be to serve python through a native web server such as tornado or gunicorn and use apache as a reverse proxy, which may improve the backend response although still won't help if processes are causing large numbers of waiting threads.

Simon Greenwood
  • 1,363
  • 9
  • 12
  • "Your httpd worker threads are therefore in a wait state and will build up as requests come in while other requests are being processed, so even a static page will end up waiting if your threads are occupied." So the question is about optimal combination for this situation. I want static pages (pages without calculations) to get very fast and pages with calculations are already loaded asynchronously. – Paul R Feb 07 '18 at 16:27
  • See the last paragraph of my reply. You can also route with apache as long as your static pages are identifiable, such as ending in `.html` but you would need separate processes to handle static dynamic objects and the response from `mod_wsgi` if the dynamic handler is still blocking. However, if as you say, your calculations are handled asynchronously then you need to look at a different way to handle the responses on the page. – Simon Greenwood Feb 07 '18 at 16:57
  • Can you provide some example configuration? – Paul R Feb 07 '18 at 17:14
  • In all honesty the simplest answer would probably to serve static objects from another cloud instance. However, you need to stop your application causing httpd threads to wait as no amount of tweaking will stop that, and the answer is most probably to use a browser based front end to poll for responses from the back. – Simon Greenwood Feb 07 '18 at 22:26
  • What is the difference between Apache number of processes and mod_wsgi number of processes? How do they interact? – Paul R Feb 08 '18 at 12:14
  • `mod_wsgi` calls python to process a request from an apache thread so there would usually be a 1:1 correlation. As an alternative you could consider a python web server such as `gunicorn` and use apache as a reverse proxy which may improve performance. – Simon Greenwood Feb 08 '18 at 14:01