0

I have an AWS server which is setup with nginx with Magento, and PHP-FPM.

The server is currently setup as an on-demand type (which I realize is the wrong type to use).

Type: c3.2xlarge     
CPU : 8  (2.80GHz)
RAM : 16GB
Storage : 1TB
Php version : 5.3.29
Database: Mysql 5.5.42
Magento : Community Edition 1.7.0.2

We are seeing very poor performance of the website generally throughout the day, but at certain times it gets even worse, with CPU usage of all 8 core processors touching and remaining at 100%.

This is a typical bad result:

http://www.webpagetest.org/result/151210_YE_WGT/

And the 'top' command output at that time:

top - 16:14:39 up 23:56,  4 users,  load average: 16.79, 12.53, 9.55
Tasks: 160 total,  18 running, 142 sleeping,   0 stopped,   0 zombie
Cpu(s): 95.4%us,  3.7%sy,  0.0%ni,  0.2%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
Mem:  15406732k total, 15262420k used,   144312k free,  1488728k buffers
Swap:  2097148k total,     5848k used,  2091300k free, 10431604k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10529 apache    20   0  585m  97m  60m R 46.5  0.7   3:09.96 php-fpm
10494 apache    20   0  503m  88m  58m R 44.2  0.6   2:40.60 php-fpm
10562 apache    20   0  579m  75m  55m R 43.9  0.5   2:37.47 php-fpm
10566 apache    20   0  503m  88m  59m R 43.9  0.6   2:06.87 php-fpm
10570 apache    20   0  508m  95m  61m R 43.9  0.6   2:58.97 php-fpm
10541 apache    20   0  505m  82m  51m R 43.6  0.5   3:06.50 php-fpm
10549 apache    20   0  508m  90m  56m R 43.6  0.6   2:51.53 php-fpm
10552 apache    20   0  501m  83m  57m R 43.2  0.6   3:07.36 php-fpm
10577 apache    20   0  505m  87m  56m R 43.2  0.6   2:53.10 php-fpm
10576 apache    20   0  505m  76m  45m R 42.9  0.5   2:33.44 php-fpm
10559 apache    20   0  506m  90m  58m R 41.9  0.6   3:05.21 php-fpm
10554 apache    20   0  503m  91m  61m S 40.9  0.6   2:43.36 php-fpm
10572 apache    20   0  502m  82m  53m R 40.6  0.5   2:59.19 php-fpm
10567 apache    20   0  503m  79m  50m R 39.6  0.5   2:05.11 php-fpm
10553 apache    20   0  504m  84m  54m R 38.9  0.6   2:41.71 php-fpm
10535 apache    20   0  506m  91m  60m R 34.6  0.6   3:09.11 php-fpm
10571 apache    20   0  569m  85m  61m S 34.6  0.6   3:02.31 php-fpm
10516 apache    20   0  496m  82m  60m S 33.6  0.6   2:29.04 php-fpm
10573 apache    20   0  500m  85m  58m R 29.3  0.6   2:30.34 php-fpm
10546 apache    20   0  501m  80m  52m R 19.6  0.5   2:40.74 php-fpm
 1536 root      20   0 1089m 743m 1848 S  1.3  4.9  11:51.52 redis-server
 2722 do_ssh_a  20   0  117m 3716 2768 S  1.0  0.0  12:38.60 htop
 3776 nobody    20   0 74312  11m 5392 S  1.0  0.1   0:12.81 nginx
17658 do_ssh_a  20   0  119m 5508 2768 S  1.0  0.0   6:50.49 htop
 3775 nobody    20   0 74244  11m 5392 S  0.3  0.1   0:12.35 nginx

The Time To First Byte (TTFB) shoots up to 27 seconds, from a "good" webpagetest result of 4 seconds (http://www.webpagetest.org/result/151210_MV_VQ3/).

The nginx configuration was earlier:

pm = static
pm.max_children = 20

We tried

pm = ondemand
pm.process_idle_timeout = 10s;

But that didnt help much. Any inputs on what I should look at to investigate this further?

EDIT1

I made the following change in /etc/php-fpm.d/www.conf 

pm = ondemand
pm.process_idle_timeout = 30s;
pm.max_requests = 50

Which greatly improved the load times.

From:

Load Time   First Byte  
37.847s     33.165s
(http://www.webpagetest.org/result/151211_EX_H2C/)

To:

Load Time   First Byte  
10.203s     5.141s
(http://www.webpagetest.org/result/151211_GX_J5N/)

EDIT2

'top' output after changes mentioned in EDIT1

top - 11:59:21 up 1 day, 19:40,  6 users,  load average: 12.79, 12.09, 12.73
Tasks: 184 total,  13 running, 170 sleeping,   0 stopped,   1 zombie
Cpu(s): 93.7%us,  3.9%sy,  0.0%ni,  1.7%id,  0.0%wa,  0.0%hi,  0.7%si,  0.0%st
Mem:  15406732k total, 15202804k used,   203928k free,  1238548k buffers
Swap:  2097148k total,     7580k used,  2089568k free, 10513760k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6585 apache    20   0  503m  77m  47m S 64.4  0.5   0:45.25 php-fpm
 6587 apache    20   0  506m  90m  58m R 52.5  0.6   0:39.24 php-fpm
 6580 apache    20   0  510m  88m  52m R 51.8  0.6   0:50.26 php-fpm
 6581 apache    20   0  505m  79m  47m R 50.5  0.5   0:45.86 php-fpm
 6570 apache    20   0  511m  95m  58m R 49.8  0.6   0:48.71 php-fpm
 6682 apache    20   0  506m  77m  44m R 49.5  0.5   0:19.54 php-fpm
 6608 apache    20   0  513m  97m  57m R 48.8  0.6   0:41.21 php-fpm
 6584 apache    20   0  505m  85m  54m S 48.5  0.6   0:43.91 php-fpm
 6613 apache    20   0  504m  78m  47m R 48.5  0.5   0:37.55 php-fpm
 6616 apache    20   0  594m  83m  48m R 47.8  0.6   0:36.26 php-fpm
 6615 apache    20   0  503m  80m  51m R 41.5  0.5   0:30.82 php-fpm
 6586 apache    20   0  506m  87m  55m R 34.9  0.6   0:41.42 php-fpm
 6576 apache    20   0  498m  75m  50m S 33.2  0.5   0:52.00 php-fpm
 6609 apache    20   0  497m  84m  61m S 28.6  0.6   0:36.51 php-fpm
 6588 apache    20   0  588m  80m  51m S 26.9  0.5   0:40.33 php-fpm
 6575 apache    20   0  582m  75m  52m R 20.9  0.5   0:49.75 php-fpm
 6612 apache    20   0  504m  85m  55m S 20.6  0.6   0:37.31 php-fpm
 6614 apache    20   0  582m  75m  52m S 20.3  0.5   0:33.36 php-fpm
 6579 apache    20   0  496m  77m  54m R 19.9  0.5   0:49.18 php-fpm
 6577 apache    20   0  499m  82m  57m S 19.3  0.6   0:50.45 php-fpm
21731 do_ssh_a  20   0  119m 5264 2620 S  1.7  0.0   5:44.07 htop
 3800 ec2-user  20   0  117m 3340 2628 S  1.3  0.0   0:36.39 htop
 6527 do_ssh_a  20   0  117m 3464 2636 S  1.3  0.0   0:03.67 htop
 1536 root      20   0 1257m 738m 1820 S  1.0  4.9  21:07.51 redis-server
 3079 nobody    20   0 73632  10m 5592 S  0.7  0.1   0:06.75 nginx
siliconpi
  • 1,807
  • 6
  • 32
  • 46
  • Please collect performance data from your server (CPU usage, what processes are using CPU, RAM usage, disk IO, etc.) and post them and your question will be re-opened. As it stands, there's not enough information for anyone to help you. – EEAA Dec 10 '15 at 23:57
  • Hi EEAA, thank you for your comments - I've added the top command output. Is that fine in your opinion? I was not able to capture more during the "heavy" load. – siliconpi Dec 11 '15 at 00:08
  • You appear to have 8 CPU cores but enough load to keep 16 occupied. Either some of your php code is implemented inefficiently or you simply need more CPU cores. I would first look for tools to profile the php code in order to identify where it is spending the CPU time. I would also add `%D` to the `LogFormat` in the apache configuration, such that you will know which requests are slow. – kasperd Dec 11 '15 at 09:52
  • Hi @kasperd - thanks for replying. Could you suggest a good profiler that I could use? – siliconpi Dec 11 '15 at 11:49
  • @siliconpi I never worked with a php project large enough to need profiling. – kasperd Dec 11 '15 at 12:19
  • You should really take a look at blackfire.io which is an excellent profiler for Php/Magento – Willem Dec 12 '15 at 00:53
  • Thank you Willem - I'm looking at exactly that now! Appreciate your and kasperd's help! – siliconpi Dec 12 '15 at 05:32
  • Please also have a look at EDIT1. I made those changes and saw some immediate improvement. 'top' output in EDIT2. – siliconpi Dec 12 '15 at 06:10

0 Answers0