VPS 512 MB RAM with WordPressMU comes to consumes lots of memory

Question

I have googled for days and gathered all optimization suggestions and tried.
My sites are not getting any high hits.
May be like 100 hits per day [all my sites combined].

Here are my specs

I have 512 MB RAM VPS with burstable 1024 MB.

Centos 5 32-bit & cPanel/WHM
Apache 2.2
MySQL 5.0
PHP 5.3.2

Here is my Configs

I have 2 WordPressMU production sites, and 1 test site

my.cnf

I used the my-medium.cnf sample to start off

# The following options will be passed to all MySQL clients
[client]
#password   = your_password
port        = 3306
socket      = /var/lib/mysql/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port        = 3306
socket      = /var/lib/mysql/mysql.sock
skip-locking
skip-bdb
skip-innodb
key_buffer = 16M
max_allowed_packet = 1M
table_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

#CAPitalZ
thread_cache_size=8
thread_concurrency=4
#query_cache_type=1
#query_cache_limit=1M
query_cache_size=16M
concurrent_insert=2
low_priority_updates=1
max_connections=50
tmp_table_size=16M
max_heap_table_size=16M
join_buffer_size=1M
interactive_timeout=25
wait_timeout=1000
#connect_timout=10 not able to restart mysql
max_connect_errors=10

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
# 
skip-networking

# Disable Federated by default
skip-federated

# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id   = 1

[mysqld_safe]
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

httpd.conf

I have unselected many modules and recompiled using EasyApache in WHM.
Only have the following modules built

Deflate
Expires
Fileprotect
Imagemap
MPM Prefork
Version [default]
EAccelerator for PHP
Bcmath
Calendar
CurlSSL [I'm using Curl.  But I don't have any https sites]
Expat 
GD    [for image cropping]
Gettext 
Imap 
Mbregex [default]
Mbstring [need both Mbregex and Mbstring for utf-8]
Mysql of the system
MySQL "Improved" extension.
Sockets 
TTF (FreeType) [I'm using custom font]
Zlib 

Under Global Configuration

I only have FollowSymLinks enabled
I Have TraceEnable, ServerSignature, FileETag OFF
ServerTokens ProductOnly
DirectoryIndex Priority has index.php as the first one

I have removed Clamd [Clam Anti-virus]

SpamAssasin is Off

Under Tweak Settings

Default catch-all/default address behavior for new accounts.  This is set to "fail"
All stats programs turned off

I have eAccelerator installed and checked in phpinfo and its working

[Pre VirtualHost Include under WHM]

Timeout 20
KeepAlive On
MaxKeepAliveRequests 200
KeepAliveTimeout 3
MinSpareServers 1
MaxSpareServers 3
StartServers 1
ServerLimit 50
MaxClients 50
MaxRequestsPerChild 4000

ExtendedStatus Off

#ServerType standalone this throws error
HostnameLookups Off

<Directory "/">
AllowOverride None
</Directory>

My sites will take ages to load and WHM/CPanel will not even load.
adadaa.com/
adadaa.net/
kadais.ca/

My average memory consumption is like 1000 MB! [yes always bursting]
The process that consumes most CPU and also most memory is mysql
But I also get like 15 httpd processes [when its bursting]
I already got warning from cpuwatchcheck saying
"While processing, the cpu has been maxed out for more than a 6 hour period. The current load/uptime line on the server at the time of this email is 07:00:37 up 11:30, 0 users, load average: 14.64, 16.79, 20.07"

I don't know, I have tried switching these config values many different times, but nothing seems to work.

Please show some light... Thanks

Here is show processlist of mysql when RAM is 800MB

Here is the Current Running Processes from WHM when RAM is around 800 MB

Pid     Name    File    Current Directory   Command Line ▾
11758   (httpd)         
24235   (mysqld_safe)   /bin/bash   /   /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/us.adadaa.org.pid
32254   (udevd) /sbin/udevd /   /sbin/udevd -d
3591    (simpleps)  /usr/local/bin/perl /usr/local/cpanel/whostmgr/docroot  /usr/bin/perl /scripts/simpleps --html
1393    (httpd) /usr/local/apache/bin/httpd /   /usr/local/apache/bin/httpd -k start -DSSL
2005    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
2012    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
3399    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
3420    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
3472    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
3493    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
3494    (httpd) /usr/local/apache/bin/httpd /   /usr/local/apache/bin/httpd -k start -DSSL
3872    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
7418    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
9482    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
9900    (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
13466   (httpd) /usr/local/apache/bin/httpd /   /usr/local/apache/bin/httpd -k start -DSSL
13656   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
13736   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
14226   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
15579   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
15623   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
15685   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
15808   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
15814   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
15919   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
16156   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
17526   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
17848   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
18159   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
19821   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
19830   (httpd) /usr/local/apache/bin/httpd /   /usr/local/apache/bin/httpd -k start -DSSL
19962   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
19980   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
19981   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
20378   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
21595   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
22513   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
23960   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
23965   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
25862   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
26305   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
26339   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
26472   (httpd) /usr/local/apache/bin/httpd /   /usr/local/apache/bin/httpd -k start -DSSL
27890   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
28600   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
29712   (httpd) /usr/local/apache/bin/httpd /   /usr/local/apache/bin/httpd -k start -DSSL
30478   (httpd) /usr/local/apache/bin/httpd /home/ada/public_html   /usr/local/apache/bin/httpd -k start -DSSL
1408    (leechprotect)  /usr/local/bin/perl /usr/local/cpanel/bin   /usr/local/cpanel/bin/leechprotect
1380    (dovecot)   /usr/sbin/dovecot   /var/run/dovecot    /usr/sbin/dovecot
1368    (exim)  /usr/sbin/exim  /var/spool/exim /usr/sbin/exim -bd -q180m
24275   (mysqld)    /usr/sbin/mysqld    /var/lib/mysql  /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/us.adadaa.org.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
32657   (named) /usr/sbin/named /var/named  /usr/sbin/named -u named
1868    (portsentry)    /usr/sbin/portsentry    /   /usr/sbin/portsentry -tcp
1411    (pure-authd)    /usr/sbin/pure-authd    /   /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth
1844    (saslauthd) /usr/sbin/saslauthd /var/run/saslauthd  /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
1845    (saslauthd) /usr/sbin/saslauthd /var/run/saslauthd  /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2
11622   (sshd)  /usr/sbin/sshd  /   /usr/sbin/sshd
1668    (cpanellogd)    /usr/local/bin/perl /   cpanellogd - sleeping for logs
1645    (cpsrvd-ssl)    /usr/local/cpanel/cpsrvd-ssl    /usr/local/cpanel/base  cpsrvd - waiting for connections
1422    (crond) /usr/sbin/crond /var/spool  crond
1381    (dovecot-auth)  /usr/libexec/dovecot/dovecot-auth   /var/run/dovecot    dovecot-auth
1397    (imap-login)    /usr/libexec/dovecot/imap-login /var/run/dovecot/login  imap-login
1398    (imap-login)    /usr/libexec/dovecot/imap-login /var/run/dovecot/login  imap-login
1   (init)  /sbin/init  /   init [3]
1395    (pop3-login)    /usr/libexec/dovecot/pop3-login /var/run/dovecot/login  pop3-login
1396    (pop3-login)    /usr/libexec/dovecot/pop3-login /var/run/dovecot/login  pop3-login
1409    (pure-ftpd) /usr/sbin/pure-ftpd /   pure-ftpd (SERVER)
1592    (queueprocd)    /usr/local/bin/perl /   queueprocd - wait to process a task
32645   (syslogd)   /sbin/syslogd   /   syslogd -m 0
1618    (tailwatchd)    /usr/local/bin/perl /   tailwatchd
3589    (whostmgr)  /usr/local/cpanel/whostmgr/bin/whostmgr /usr/local/cpanel/whostmgr/docroot  whostmgr - simpleps ./simpleps
3520    (cpsrvd-ssl)    /usr/local/cpanel/cpsrvd-ssl    /usr/local/cpanel/whostmgr/docroot  whostmgrd - serving 207.219.51.3
3525    (cpsrvd-ssl)    /usr/local/cpanel/cpsrvd-ssl    /usr/local/cpanel/whostmgr/docroot  whostmgrd - serving 207.219.51.3
32690   (xinetd)    /usr/sbin/xinetd    /   xinetd -stayalive -pidfile /var/run/xinetd.pid 

================

Here is the Current CPU usage when around 800 MB RAM usage

Pid Owner   Priority    Cpu %   Mem % ▴ Command
 25862   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.6     7.5    
 26305   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.3     6.5    
 30478   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     6.2    
 14226   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.1     6.1    
 28600   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.1     6.0    
 24275   mysql   0           /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/us.adadaa.org.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock 
             30.6    5.8    
 19980   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.1     3.2    
 13656   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.1     2.9    
 15685   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             1.6     2.9    
 2005    nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.0     2.8    
 20378   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.8    
 15579   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.8    
 18159   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.6     2.8    
 27890   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.0     2.7    
 22513   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.7    
 26339   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.4     2.7    
 13466   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.4     2.7    
 3872    nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.6    
 15623   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.6    
 17526   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.0     2.5    
 7418    nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.0     2.5    
 23960   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.1     2.5    
 13914   root    0           whostmgr2 - top ./top  
             0.2     2.5    
 12035   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.3     2.5    
 21595   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.3     2.5    
 13736   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.4    
 16156   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.4    
 15808   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.4    
 13557   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.6     2.4    
 3493    nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.0     2.3    
 23965   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.1     2.3    
 9900    nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.3    
 19962   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.2     2.3    
 12026   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.4     2.1    
 19821   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.0     2.0    
 15919   nobody      0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.1     2.0    
 1393    root    0           /usr/local/apache/bin/httpd -k start -DSSL 
             0.0     1.0    
 13689   root    0           whostmgrd - serving 207.219.51.3 
             0.0     0.9    
 13898   root    0           whostmgrd - serving 207.219.51.3 
             0.0     0.8    
 1645    root    0           cpsrvd - waiting for connections 
             0.0     0.7    
 1408    root    0           /usr/local/cpanel/bin/leechprotect 
             0.0     0.4    
 32657   named   0           /usr/sbin/named -u named 
             0.0     0.3    
 1592    root    0           queueprocd - wait to process a task 
             0.0     0.3    
 13698   root    0           tailwatchd 
             0.0     0.3    
 1618    root    0           tailwatchd 
             0.0     0.2    
 1668    root    18              cpanellogd - sleeping for logs 
             0.0     0.1    
 24235   root    0           /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/us.adadaa.org.pid 
             0.0     0.1    
 1422    root    0           crond  
             0.0     0.1    
 1411    root    0           /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth 
             0.0     0.1    
 1409    root    0           pure-ftpd (SERVER)  
             0.0     0.1    
 1398    dovecot     0           imap-login  
             0.0     0.1    
 1397    dovecot     0           imap-login  
             0.0     0.1    
 1396    dovecot     0           pop3-login  
             0.0     0.1    
 1395    dovecot     0           pop3-login  
             0.0     0.1    
 1368    mailnull    0           /usr/sbin/exim -bd -q180m 
             0.0     0.1    
 11622   root    0           /usr/sbin/sshd 
             0.0     0.1    
 32690   root    0           xinetd -stayalive -pidfile /var/run/xinetd.pid  
             0.0     0.0    
 32645   root    0           syslogd -m 0  
             0.0     0.0    
 32254   root    -4              /sbin/udevd -d 
             0.0     0.0    
 1868    root    0           /usr/sbin/portsentry -tcp 
             0.0     0.0    
 1845    root    0           /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2 
             0.0     0.0    
 1844    root    0           /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2 
             0.0     0.0    
 1381    root    0           dovecot-auth  
             0.0     0.0    
 1380    root    0           /usr/sbin/dovecot 
             0.0     0.0    
 14050   root    0           /bin/ps -ewwo pid,user,nice,pmem,pcpu,command 
             0.0     0.0    
 1   root    0           init [3]  
             0.0     0.0    

Answer 1

I wonder if anything interesting is showing up in your log files (perhaps under /var/log/httpd ?

It almost looks like you moved things out of .htaccess into an /etc/httpd/conf file, and that can help keep apache from groping the directories for an .htaccess file.

Can you log in via ssh or putty somehow and get a command shell where you can run top?

One gig is a small amount of memory, but I'm running about 5 wpmu sites and a tikiwiki site under 256meg with some playing around with PHPCake and Ruby on Rails. I was thinking to move over some Drupal sites I have as well, but I'm running a little low on memory.

I also have an APC cache, but I use it strictly for code caching and keep the page and object caching on disk, though I am using W3 Total Cache for my WPMU sites rather than Super Cache which I also found to be quite good.

I'm not using any panels on my VPS system. I like the freedom of working directly in Linux. I'm running exim for my email server and dovecot for pop3. I'd like to get python and tomcat set up as well as I have a sort of MVC unification kind of idea I'd like to look into whether it would be an idea with potential or not.

Anyway, my wpmu sites are small, but my VPS only has a quarter of what yours apparently has. Come to think of it, one of the first things I noticed when bringing the system up is that I had script kiddies and spam bots hitting my VPS before I even had it on DNS! And some were doing some brute force attacks, so I set up some intrusion detection and dropped them into the firewall for a couple days, and others permanently. I also installed a lot of security into the websites through apache and through wordpress plugins to bat away sql insertion and junk.

I wonder if you're getting some brute force attacks coming in through SSL with 50 apache lines open, though I noticed you mentioned there were about 15 concurrent connections perhaps?

Running top, ps, vmstat, iostat, or whatever you have available might be able to give you an idea of where the bottlenecks are. Is your paging or scan rate heavy? Do you have a huge number of runnables? What's the CPU load? If you're compressing and encrypting a zillion things per second on the fly all the time, your cpu load would easily go through the stratosphere.

Or maybe one of your pages became popular on Digg? Sorry. It's late and I probably should get some sleep as I'm wandering all over the place, and I don't know if I'm hitting on anything that might give you any valuable ideas....

Best regards, Dan