0

LAMP Server Irregular out of memory. Need help to find out what it cause.

When it happen, unable SSH to the server. Restart httpd or mysqld cannot resolve the issue except fully stop the server and start the server from AWS Console. Reboot sometime will help but mostly not.

Web Application: Drupal 7
Server Spec: AWS EC2, 2 core, 4 GB Memory
Stack: LAMP stack (AWS Linux, Apache2.4, MySQL 5.5, PHP5.6)
Note:
- No Swap
- This case never happen since server launch for 1 year until recently
- No major change on the server and web application except OS security patch and drupal security patch

https://i.stack.imgur.com/cbv8e.png

As per graph above, it hit the max of memory irregular with different timeframe. To further prove it is irregular, 2 test had been conducted:

Test 1: Clone the server with same spec and let it run concurrent with production server. Cloned server no public visitation.
Result: Production server still encounter irregular out of memory but cloned server never hit more than 35% memory

Test 2: Switch cloned server as production server (public access)
Result: Will encounter irregular out of memory too

From the test, 2 identical server with same recurring tasks, cron jobs, etc, but only server with public access will hit out of memory.

Message Log: https://i.stack.imgur.com/7sAKV.png
MySQL Log 1: https://i.stack.imgur.com/Vy4Kn.png
MySQL Log 2: https://i.stack.imgur.com/mtp6z.png
Access Log: https://i.stack.imgur.com/d8BJZ.png
Nothing identical issue. Internal dummy connection is AWS Health check. Without out of memory also similar log.
Error Log: https://i.stack.imgur.com/mK3Nm.png
Nothing identical error found but just out of memory notice
Drupal Log: Nothing logged during put of memory period

--- Additional Information 13 July 2018 ---
Top command during high memory utilisation
https://i.stack.imgur.com/pxCuX.png

Top command after 5 minute server totally reboot
https://i.stack.imgur.com/yxWDv.png

Notes:
- After reboot, it will never hit again at least 12 - 24 hour

--- Additional Information 25 July 2018 ---
Average apache per thread is 2%, and calculate the memory usage and adding apache configuration on the memory usage as per below. The server will not hit more than 70% memory, but server will freeze and and no reponding until stop and start the server.

StartServers 3 MinSpareServers 3 MaxSpareServers 5 MaxClients 36 ServerLimit 36 MaxRequestsPerChild 4500

Leong
  • 1
  • 1
  • It seems to be happening approximately daily. Look at recurring tasks, cron jobs, etc. – Michael Hampton Jul 11 '18 at 12:46
  • The reason that you have to reboot the server is that the server is failing due to failure to create new processes (threads). This is messing up either the OS or your applications running on the server. I would enable swap and then monitor how much swap space actually gets used. If it is a small amount then you may need to select an instance with more memory. Most likely you will find the problem in your web logs. A bad actor or a search engine is hitting your site and consuming resources. – John Hanley Jul 12 '18 at 06:52

0 Answers0