3

On a "CentOS release 6.2 (Final)" server (with Ruby 1.9.3 and Rails 3.2), and using more memory than expected.

Looking at passenger-memory-stats I see a couple of HUGE httpd processes... any thoughts on how I can figure out what's going on and reduce the memory usage?

Stats are included here... thanks!

---------- Apache processes -----------
PID    PPID  VMSize     Private   Name
---------------------------------------
1371   1     202.1 MB   0.1 MB    /usr/sbin/httpd
4573   1371  210.2 MB   5.0 MB    /usr/sbin/httpd
4778   1371  202.5 MB   0.6 MB    /usr/sbin/httpd
4780   1371  217.6 MB   9.4 MB    /usr/sbin/httpd
4781   1371  217.1 MB   9.1 MB    /usr/sbin/httpd
4856   1371  202.4 MB   0.5 MB    /usr/sbin/httpd
4863   1371  204.1 MB   2.1 MB    /usr/sbin/httpd
5027   1371  202.4 MB   0.5 MB    /usr/sbin/httpd
5043   1371  202.4 MB   0.4 MB    /usr/sbin/httpd
5044   1371  205.5 MB   2.7 MB    /usr/sbin/httpd
5072   1371  202.4 MB   0.5 MB    /usr/sbin/httpd
5084   1371  202.4 MB   0.5 MB    /usr/sbin/httpd
32111  1371  1297.0 MB  246.5 MB  /usr/sbin/httpd
32579  1371  1914.3 MB  215.5 MB  /usr/sbin/httpd
### Processes: 14
### Total private dirty RSS: 493.42 MB


-------- Nginx processes --------

### Processes: 0
### Total private dirty RSS: 0.00 MB


----- Passenger processes -----
PID    VMSize    Private  Name
-------------------------------
4180   280.5 MB  24.4 MB  Passenger ApplicationSpawner: /var/www/apps/people/current
4345   309.5 MB  53.4 MB  Rack: /var/www/apps/people/current
4800   300.2 MB  55.2 MB  Rack: /var/www/apps/people/current
4808   297.8 MB  52.5 MB  Rack: /var/www/apps/people/current
4815   297.4 MB  52.4 MB  Rack: /var/www/apps/people/current
4822   302.7 MB  55.6 MB  Rack: /var/www/apps/people/current
22780  209.0 MB  0.0 MB   PassengerWatchdog
22783  991.5 MB  1.3 MB   PassengerHelperAgent
22785  113.4 MB  1.1 MB   Passenger spawn server
22788  144.6 MB  0.0 MB   PassengerLoggingAgent
22911  310.4 MB  64.0 MB  Rack: /var/www/apps/people/current
22939  311.6 MB  53.5 MB  Rack: /var/www/apps/people/current
26175  304.1 MB  55.8 MB  Rack: /var/www/apps/people/current
26182  310.4 MB  44.0 MB  Rack: /var/www/apps/people/current
### Processes: 14
### Total private dirty RSS: 513.24 MB

* UPDATE * : I realize that the bigger culprit is likely our application, rather than Rails, but am trying to figure out what's going on here in all cases... so... here's a few more specific questions:

  1. what is going on with those two huge httpd processes? I expected two large spawner processes from Passenger, but why's apache look like this?

  2. given that I need to do some memory usage profiling of our Rails application, what tools are out there to help with that? I know of lots of performance profiling, benchmarking, etc. tools - but, what is available to help tell where the memory is being used?

Brent Faulkner
  • 31
  • 4
  • Where the memory is being used is hard to say without a vmcore i.e. memory dump of the kernel. But I guess you are not willing to do that, nor that people would freely check that or you would want to give such sensitive data. Barring that, you can run a pmap -x to get the memory foot print of the process and a sysrq-t to know at what functions those processes are eating memory. Also, if you indeed think it is a memory leak or something nasty, then valgrind would help but that is again another story. – Soham Chakraborty Oct 12 '12 at 10:51
  • Do you also run mod_php/mod_perl/mod_python or anything like that? If PHP/Perl/Python scripts run through those modules leak memory, the leaked memory stays behind in the Apache processes. There are currently no known memory leaks in Phusion Passenger - it runs almost everything out-of-process specifically to avoid problems like this. – Hongli Lai Oct 24 '12 at 00:51
  • no... nothing other than the rails application – Brent Faulkner Oct 28 '12 at 03:06
  • regarding a leak... it doesn't seem to *grow* to that point... it gets there very quickly and then stays like that – Brent Faulkner Oct 28 '12 at 03:07

1 Answers1

0

Those processes are all running your Rails application. They're all relatively consistent in size, and I suspect if you included your Apache/Passenger configuration files (or at least parts of them) we'd see that the count lines up with the number of child processes you've configured. I would look first at the Rails app to see why it's using so much RAM, and then come back to Apache/Passenger.

pjmorse
  • 1,550
  • 1
  • 17
  • 34
  • however, look at those two apache processes... 32111 1371 1297.0 MB 246.5 MB /usr/sbin/httpd 32579 1371 1914.3 MB 215.5 MB /usr/sbin/httpd almost 2GB each... I expected a couple of big passenger processes for the spawners, but 4G stuck in Apache was a surprise... what's up with that? – Brent Faulkner Oct 12 '12 at 10:25
  • Good questions. Off the top of my head I don't have answers but your edits to the question should help. – pjmorse Oct 12 '12 at 10:51