5

I am running a high traffic website on a Dedicated-Virtual 16GB RAM CentOS server hosted by Media Temple.
Very often the server dies because of high CPU usage (%1500)
When i check the running processes using "top" command i see a too many httpd processes running which i think are causing the high usage of CPU.
I have tried to tune mysql, tune apache, my.cnf, httpd.cnf, removed Apache modules that i do not use but that does not solve the problem.
Please let me know what values do you need to know in my server config in order to help me diagnosis the problem.
Thanks in advance.

  • you can't have 1500% CPU without at least 15 processors - how are you determining this? –  Oct 23 '11 at 07:35
  • 1500% as per of Media Template control panel status –  Oct 23 '11 at 07:52
  • Here is the top 15 lines of 'top' command pastebin.com/5c16JN4G –  Oct 23 '11 at 07:56
  • yeah, what does top say? If you hit 'I' it switches between IRIX mode which shows 100% per core, so it can be higher than 100%, but you're probably seeing load average not CPU. –  Oct 23 '11 at 07:56
  • Hrm, so yes, all your httpds are running hot and using a full core each. –  Oct 23 '11 at 08:05
  • 1
    How many is "too many"? On a 16GB box you might be able to run a thousand httpd instances without causing trouble if it doesn't do much else. – ceejayoz Oct 23 '11 at 18:46
  • 1
    Can you access the Apache server-status page to see what all those httpds are doing? Is anything showing up in your error log? – MarkR Oct 24 '11 at 13:45

4 Answers4

7

what does tail -f /var/log/messages say when you experience high system loads?

Are your processes very I/O or very CPU intensive?

One observation:

Looks like you have 18G physical memory, but zero swap space..., e.g. your top command shows "Swap: 0k total" ... that means that you do not have any Swap Space configured.

In general, you should always have a sufficient swap space on a UNIX system! Swap-size = 1...2-times RAM size is a good idea. Using a fast partition is a good idea. Really bad things happen if your UNIX system runs out of RAM and doesn't have Swap .. processes just die inexplicably.. that is a very bad thing! especially in production. Disk is cheap! add a generous swap partition! :-) or worst case you can also create a swap-file later.

On any UNIX system, you do need swap space -- because that is where processes live while they are not scheduled on one of the CPUs. If you don't have swap space, really bad things will happen, because your system will run out of places (RAM) where to create new processes -- it can not put them in swap, because there is none, so you will see extremely high loads, the system freezing up, and processes die seemingly inexplicably.

To check your memory and swap space in use, you can run free -k. You can also check the swap configuration by running swapon -s .. this will not show any output if no swap is configured.

As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory. I'd say if you have 18G RAM, configure ~20..30G Swap -- don't be stingy with the Swap, because "disk is dirt-cheap"!

Disclaimer: there are a few exceptions when you can live without swap, but on the other hand, configuring and not needing it does not hurt! :)

Two choices for creating swap space:

  • either create a swap partition, preferably on your fastest disk partition (best choice; or add a flash drive to your physical server)

  • or create a swap file (if you don't have disk space left to partition; this is also a good option if you later find out that you need more swap than you anticipated)

...then format that swap partition or swap file to make it usable for swapping with the mkswap command.

Once you enable the swap space with sudo swapon -a, you should see it listed in top and with swapon -s , and your system should behave much more nicely..

Articles about Swap Space:

https://www.linux.com/news/software/applications/8208-all-about-linux-swap-space

https://www.linux.com/learn/tutorials/442430-increase-your-available-swap-space-with-a-swap-file

http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/

http://lissot.net/partition/partition-08.html

http://tldp.org/LDP/sag/html/swap-space.html

Some pointers:

man -k swap

man mkswap

man fstab , vim /etc/fstab

EDIT

as this question was migrated to ServerFault .. I'll add a comment here, because ppl are complaining that I quote the old rule: " 1..2-times RAM size == Swap size "

A 2TB SATA drive costs around $70 these days, that means 20GB costs you about 70 cents -- it's dirt cheap! I know 20GB sounds a lot, but think of it as a very inexpensive "worst case insurance"!

Somebody said: "if your system filled up 18GB and is swapping onto 20GB, then you have bigger problems"

Certainly true, but having generous Swap Space if unexpected things go wrong, will save your server and the processes on it from crashing completely! You might still be able to log in, examine the system while it's still running (slowly), and repair things (albeit slowly), and not have to reboot.. On a production server, there is really no reason not to use generous swap.

If I had a server with 18GB RAM and it's running MongoDB for example (which is very RAM intensive) .. I would certainly configure very generous Swap Space, perhaps even 40..60GB ... just in case something unexpected goes wrong..

Tilo
  • 391
  • 1
  • 5
  • Here is my top http://pastebin.com/GSzDhZ22 –  Oct 23 '11 at 08:03
  • 1
    "As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory." .. this was true about ten years ago. Today, if you have 20 GB of swap in active use, your server probably consumes more time in transferring stuff from and to swap, and has less time for actual work. – Janne Pikkarainen Oct 24 '11 at 09:01
  • Great answer, and generally true, but it should be clear that this remains a generalization. There are some environments where you can safely, through various controls, deploy a unix-based server and not have any swap. I've done this many times, though admittedly never in an environment where a daemon like apache httpd was running, as it spawns itself to take on new requests which is more challenging to manage. – sandroid Oct 24 '11 at 13:44
  • @sandroid : yes, certainly true! I just wanted to point out to user sunn that disk space is ridiculously cheap, and there's really no good reason to not configure swap in a regular server or desktop system -- even if it's 20GB (which sounds a lot, but it's dirt cheap). – Tilo Oct 25 '11 at 00:57
  • @JannePikkarainen : yes, you're absolutely right! The 1..2 times rule is old .. but think about how dirt cheap disk space is -- a 2TB disk costs ~$70 these days .. .that's 70 cents for the 20GB! There's no good reason to not add swap -- even if it's 20GB -- it's sort of a "worst-case insurance" when unexpected things happen.. – Tilo Oct 25 '11 at 01:01
  • I prefer OOM-Killer to do its job and kill the apache, than having a server trashing that I can not connect remotely. – Mircea Vutcovici Oct 25 '11 at 01:21
0

20GB - 30GB of swap space very high. If you are swapping 20GB in Linux you have bigger problems.

Virulence
  • 1
  • Except the top information shows zero swap-space even configured. I'm not sure where you're getting 20GB of it. – sysadmin1138 Oct 24 '11 at 11:36
  • 1
    "As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory. I'd say if you have 18G RAM, configure ~20..30G Swap " – Virulence Oct 24 '11 at 14:29
  • 20GB of swap costs you about 70 cents these days.. there is really no reason to be stingy with it – Tilo Oct 25 '11 at 01:02
0

Almost certainly your machine is running out of memory because you have too many "fat" Apache processes doing little or no work.

Apache normally uses a "prefork" model. PHP normally runs in-process. This can be problematic, because of:

  • PHP causes the apache child processes to use lots of memory each.
  • Serving static files over slow connections, keeps child processes in use for a long time for no reason
  • HTTP keepalives, if turned on, also keep child processes in use.

A "bad" solution would be to turn off keepalives (or maybe reduce the keepalive time). Note that keepalives are VERY good for performance, particularly on HTTPS.

A "better" solution would be to switch to a better architecture. A better architecture would be one where you don't waste "fat" processes doing nothing. A typical implementation of a better architecture, would be Nginx + PHP/FastCGI.

I'm not necessarily advocating the use of Nginx - you could probably do it with Apache too, if you run a different MPM (Multiprocessing module e.g. worker, event, not prefork) and move PHP out-of-process with FastCGI.

MarkR
  • 2,928
  • 17
  • 13
0

In your case you do not need swap. If you want you could create 1-2GB swap partition or file to have more physical memory for file system cache.

I do not understand why cached memory and buffers memory is 0. Can you paste the output of cat /proc/meminfo ?

You have very high I/O load. This can be disk or network. Try to reduce the load. For disk mount the data partition with noatime. Try to find out which process is most I/O intensive. Useiotopanddstat` for this.

Try to upgrade the firmware and the divers (kernel modules) for the disk controller and for the network card. If you are running a VM make sure you are installing the paravirtualization drivers (e.g. VMware Tools). If you are running a 32bit kernel with PAE, try to switch to a 64bit distribution (you will reduce from the kernel time).

Mircea Vutcovici
  • 17,619
  • 4
  • 56
  • 83
  • Please see meminfo - https://shrib.com/#BskRvetZWCeOcgQXV0MA Last few days memory issue is giving hard time. I can only see lots of httpd processes running. – Damodar Bashyal Oct 19 '19 at 09:43