1

I have strange, very strange peaks of CPU loading on my website.

normal loading about 2-3, when i have this peaks it is about 100-150 peaks may be in any time. not depends my normal loading profile.(for example at 6AM) peaks are short 2-3-5 minutes

CPU loading affect network traffic and number of nginx requests. This time i do not have many requests for web server.

Server is separated web server with nginx, php-fpm, redis, rabbit-mq, mysql on another server.

I really need advice how can i know source of this. When i will know source i will eliminate it. I will really appreciate for your advices.

Graphs from zabbix monitoring:

load peak

traffic trough

AD7six
  • 2,920
  • 2
  • 21
  • 23
andreyvlru
  • 11
  • 4
  • Honestly, it might be useful to try out a free trial of an APM tool that supports whatever language you're using on the web server. – theterribletrivium Oct 28 '14 at 00:45
  • What do you mean as APM? – andreyvlru Oct 28 '14 at 01:50
  • Sorry, I mean Application Performance Monitoring or Application Performance Management. We're not supposed to name specific brands but lots of them can be installed onto a system and will provide you with data about what exactly is running slow. – theterribletrivium Oct 28 '14 at 04:16
  • Given there is a load peak, and a traffic trough at exactly the same time - it seems quite likely the application has a bottlneck (typically access to the database) which is failing/overloaded causing everything else to wait. You need to look/find what the server was doing during the load peak. – AD7six Oct 28 '14 at 13:01
  • Looks like it is low memory issues. When php-fpm starts to use swap, we have this. – andreyvlru Oct 30 '14 at 18:26
  • I have reduced max memory for redis (more for php-fpm) and limit php-fpm max children and it becomes better. – andreyvlru Oct 30 '14 at 18:32

1 Answers1

0

It was memory issue. When memory is out, php-fpm processes freeze, going to swap and actually your webserver are stopped.

So, you should know your average memory consumption per 1 php-fpm process. Take memory allocated for all php-fpm processes at peak hours - M take total active processes - P Avg = M / P We have Avg = 30 Mb. So with php-fpm max children = 300, we need about 9 Gb memory for php-fpm. We temp fix this by decreasing memory for redis cache and will double memory for this server in future.

andreyvlru
  • 11
  • 4