26

I have been working to deploy a relatively large Rails app (Rails 2.3.5) and recently doing some load testing we discovered that the throughput for the site is way below the expected level of traffic.

We were running on a standard 32bit server, 3GB of RAM with Centos, and we were running Ruby Enterprise Edition (latest build), Passenger (Latest build) and Nginx (Latest build) - when there is only one or two users the site runs fine (as you would expect) however when we try to ramp up the load to ~50 concurrent requests it completely dies. (Apache Bench report ~2.3 req/sec, which is terrible)

We are running RPM and trying to determine where the load issue is, but it's pretty evenly distributed across Rails, SQL and Memcached, so we're more or less going through and optimizing the codebase.

Out of sheer desperation we spun up a large EC2 instance (Ubuntu 9.10, 7.5GB RAM, 2 Compute Units/Cores) and setup the same configuration as the original server, and while there are more resources we were still seeing pathetic results.

So, after spending too much time trying to optimize, playing with caching configuration etc I decided to test the throughput of some mongrels, and ta-da, they are performing much much better then Passenger.

Currently the configuration is 15x Mongrels being proxied via Nginx, and we seem to be meeting our load requirements just but it's not quite enough to make me comfortable with going live... What I was wondering is if anyone knows of some possible causes for this...?

My configuration for passenger/nginx was:

  • Nginx workers: tried between 1 and 10, usually three though.
  • Passenger max pool size: 10 - 30 (yes, these numbers are quite high)
  • Passenger global queueing: tried both on and off.
  • NGinx GZip on: yes

It might pay to note that we had increased the nginx max client body size to 200m to allow for large file uploads.

Anyway suggestions would be really appreciated, while the mongrels are working fine it changes how we do things a lot and I would really prefer to use Passenger - besides, wasn't it supposed to make this easier and perform better?

Matthew Savage
  • 3,794
  • 10
  • 43
  • 53
  • 4
    Just fyi here's the explanation: The difference is that mongrels spawn separated instances of your app so each app has its own sql pool while passenger forks new instances out of a single pre-initialized app spawner (which is much much faster) so it shares your sql pool. – hurikhan77 Feb 05 '10 at 07:50
  • Thanks for the explanation - once we tested the changes to the pools (well, once I read the suggestion) it became very very clear to me - as wise people have often said, sometimes you just need another pair of eyes ;) – Matthew Savage Feb 05 '10 at 11:07
  • You may want to amend your title as this resolved as non-nginx related. It will happen on every passenger deployment if you set the sql pool size too small. – hurikhan77 Feb 08 '10 at 16:26
  • @hurikhan77: Someone from the Phusion team [claims pools are not shared amongst apps](https://groups.google.com/forum/#!topic/phusion-passenger/hfdeaysTqPo). – twelve17 Apr 08 '14 at 12:08
  • @twelve17 It may be different now in Passenger4. But usually it goes like this (and that means not shared amongst apps but shared amonst instances of each app): Pool connections get opened, processes forked (open FDs become shared), thus pool is shared with each instance – hurikhan77 Apr 09 '14 at 19:44
  • @hurikhan77: If using the smart spawner mode, Passenger (at least as of version 3) has a mechanism that would create new connections for forked processes. See section: "12.3. Smart spawning gotcha #1: unintential file descriptor sharing": "Note that Phusion Passenger automatically reestablishes the connection to the database upon creating a new worker process, which is why you normally do not encounter any database issues when using smart spawning mode." – twelve17 Apr 24 '14 at 11:31
  • @twelve17 Well I started with Passenger 2 - which is probably around the time when this question was asked. Smart spawning was a later addition and AFAIR not compatible with all installations. – hurikhan77 Apr 28 '14 at 11:32

2 Answers2

18

Maybe your sql pool size is too small? This essentially limits the parallelism of database workloads in your application which in turn builds up to much increased load as soon as you put work on your app stack...

hurikhan77
  • 5,881
  • 3
  • 32
  • 47
  • Wow, I can't believe I didn't think of this - we jacked up the pool to a higher number (is there such thing as 'too high'? is 50 too high?) and the difference is *amazing*. Thanks so much for your suggestion – Matthew Savage Feb 05 '10 at 00:27
  • You will need no more than your passenger pool size... Maybe give an extra buffer of 4 or 5. – hurikhan77 Feb 05 '10 at 07:44
  • @hurikhan77 - any suggestions on passenger pool sizes? I guess 'it depends on the site, server, and demands...'? ;) – Matthew Savage Feb 05 '10 at 11:08
  • 4
    This depends on your memory size and how much your application uses. It should fullfill something like this: "poolsize * process_size < 80% * ram" and "accesses_per_second * response_time < poolsize", else your server won't fit or your response time would suffer. Keep in mind that this equation does not count mysql (for mysql add 2 mb per app process size + 1 or 2 gb if it runs on the same machine). – hurikhan77 Feb 05 '10 at 17:10
  • cut "size" from "2 mb per app process size" - was too late to edit my comment ;-) BTW: Process size may include nginx process size but i suppose it is multithreaded anyway? – hurikhan77 Feb 05 '10 at 17:17
  • These numbers are only a starting point. Put your application under expected loads and check the numbers. The formulas show how everything is connected to each other to help you turn the right knobs. Do not try to raise the poolsize if your app becomes slow without checking free RAM. Your server should not start swapping or your app will thrash. 32bit mysql may be a bad idea (it may become unstable as soon as it allocates >2GB ram in 32bit environment). – hurikhan77 Feb 05 '10 at 17:28
  • Why is the behaviour different in passenger compared to mongrel? In mongrel we don't need to worry so much about db pool size. – Joshua Partogi May 29 '10 at 00:13
  • 1
    This is the 'pool' parameter in the database.yml we're talking about, right? – miccet Feb 24 '11 at 15:58
  • 1
    +1 to this just got a speed increase to my app by doubling the pool size to match the passenger max pool size – Nick M Aug 03 '16 at 12:32
2

As a first step I would deploy a minimal "Hello World" type Rails application to your environment and see what throughput you get with that. Doing that will at least tell you if your problem is with the environment or somewhere in your application.

John Topley
  • 113,588
  • 46
  • 195
  • 237