Magento, NGINX, PHP-FPM, APC, MEMCACHED, 16gb Ram CentOS, Spiking PHP-FPM to 100% CPU

Question

I have been trying to resolve my issue of spiking cpu caused by php-fpm processes. I've reduced the php-fpm config settings to:

pm = ondemand
pm.max_children = 12
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 500
php_admin_value[memory_limit] = 128M

Problem still exists. I'm running a Joomla main site (which is having no problems) and a Magento store in a sub-directory. My server is a Linux CentOS, running NGINX, APC, Memcached, Full Page Cache and php-fpm. My server has 8 cores and 16gb dedicated ram. My host has shut down my server several times the past week because my php-fpm processes are consuming the entire network. A lot of the individual php-fpm processes are getting over 50% cpu. I've hired several "professionals" and none of them was able to help me, so now broke and stumped, I'm turning to you guys for help. So any suggestions would be greatly appreciated. I turned on slow php logs and here are some of the latest results:

[01-Apr-2012 14:26:12]  [pool magento] pid 21537
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000011a394f8] _renderStraightjoin() /home/flyfish/www/flyshop/lib/Varien/Db/Select.php:397
[0x0000000011a39158] _renderStraightjoin() /home/flyfish/www/flyshop/lib/Zend/Db/Select.php:705
[0x0000000011a38f30] assemble() /home/flyfish/www/flyshop/lib/Zend/Db/Select.php:1343
[0x00007fffbb6d6e50] __toString() unknown:0
[0x0000000011a38630] _prepareQuery() /home/flyfish/www/flyshop/lib/Varien/Db/Adapter/Pdo/Mysql.php:409
[0x0000000011a38270] _prepareQuery() /home/flyfish/www/flyshop/lib/Varien/Db/Adapter/Pdo/Mysql.php:388
[0x0000000011a38008] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:734
[0x0000000011a375c8] fetchAll() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Product/Type/Configurable/Attribute/Collection.php:196
[0x0000000011a370e0] _loadLabels() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Product/Type/Configurable/Attribute/Collection.php:129
[0x0000000011a369a0] _afterLoad() /home/flyfish/www/flyshop/lib/Varien/Data/Collection/Db.php:536
[0x0000000011a364a8] load() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:253
[0x0000000011a35968] getConfigurableAttributes() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:330
[0x0000000011a35590] getUsedProducts() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:458
[0x0000000011a35410] isSalable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1264
[0x0000000011a35098] isAvailable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1244
[0x0000000011a34fa8] isSalable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1308
[0x0000000011a33998] isSaleable() /home/flyfish/www/flyshop/app/design/frontend/moxy/default/template/rokmagemodules/rokmage-categoryview/rokmage-categoryview.phtml:122
[0x0000000011a331f0] +++ dump failed

[01-Apr-2012 14:26:44]  [pool magento] pid 21531
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000011a37768] _loadPrices() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Product/Type/Configurable/Attribute/Collection.php:251
[0x0000000011a37280] _loadPrices() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Product/Type/Configurable/Attribute/Collection.php:132
[0x0000000011a36b40] _afterLoad() /home/flyfish/www/flyshop/lib/Varien/Data/Collection/Db.php:536
[0x0000000011a36648] load() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:253
[0x0000000011a35b08] getConfigurableAttributes() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:330
[0x0000000011a35730] getUsedProducts() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:458
[0x0000000011a355b0] isSalable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1264
[0x0000000011a35238] isAvailable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1244
[0x0000000011a35148] isSalable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1308
[0x0000000011a33b38] isSaleable() /home/flyfish/www/flyshop/app/design/frontend/moxy/default/template/rokmagemodules/rokmage-categoryview/rokmage-categoryview.phtml:122
[0x0000000011a33390] +++ dump failed

[01-Apr-2012 14:27:01]  [pool magento] pid 21528
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000011ff67a8] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement/Pdo.php:228
[0x0000000011ff6518] _execute() /home/flyfish/www/flyshop/lib/Varien/Db/Statement/Pdo/Mysql.php:110
[0x0000000011ff5e90] _execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement.php:300
[0x0000000011ff5a20] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:479
[0x0000000011ff5438] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Pdo/Abstract.php:238
[0x0000000011ff5078] query() /home/flyfish/www/flyshop/lib/Varien/Db/Adapter/Pdo/Mysql.php:389
[0x0000000011ff4e98] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:825
[0x0000000011ff4948] fetchOne() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Category/Flat.php:1161
[0x0000000011ff4678] getProductCount() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Category.php:801
[0x0000000011ff33e0] getProductCount() /home/flyfish/www/flyshop/app/code/local/Extendware/EWLayeredNav/Model/Library/Plugin/Catalog/Layer/Filter/Category.php:54
[0x0000000011ff2da0] _initItemsData() /home/flyfish/www/flyshop/app/code/local/Extendware/EWLayeredNav/Model/Library/Plugin/Catalog/Layer/Filter/Category.php:23
[0x0000000011ff2818] _getItemsData() /home/flyfish/www/flyshop/app/code/local/Extendware/EWLayeredNav/Model/Library/Plugin/Catalog/Layer/Filter/Category.php:119
[0x0000000011ff26b0] _initItems() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Layer/Filter/Abstract.php:120
[0x0000000011ff2598] getItems() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Layer/Filter/Abstract.php:109
[0x0000000011ff2480] getItemsCount() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Block/Layer/Filter/Abstract.php:126
[0x0000000011ff22b8] getItemsCount() /home/flyfish/www/flyshop/var/cache/extendware/ewcore/overrides/Mage/Catalog/Block/Layer/View/67dcc5dfa9c44bd3a205b75a08193105.php:218
[0x0000000011ff2088] canShowOptions() /home/flyfish/www/flyshop/var/cache/extendware/ewcore/overrides/Mage/Catalog/Block/Layer/View/67dcc5dfa9c44bd3a205b75a08193105.php:233
[0x0000000011ff14f8] canShowBlock() /home/flyfish/www/flyshop/app/design/frontend/moxy/default/template/extendware/ewlayerednav/catalog/layer/view.phtml:6
[0x0000000011ff0d50] +++ dump failed

[01-Apr-2012 14:27:04]  [pool magento] pid 21529
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000012468ff8] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement/Pdo.php:228
[0x0000000012468d68] _execute() /home/flyfish/www/flyshop/lib/Varien/Db/Statement/Pdo/Mysql.php:110
[0x00000000124686e0] _execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement.php:300
[0x0000000012468270] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:479
[0x0000000012467c88] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Pdo/Abstract.php:238
[0x00000000124678c8] query() /home/flyfish/www/flyshop/lib/Varien/Db/Adapter/Pdo/Mysql.php:389
[0x0000000012467660] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:734
[0x0000000012467248] fetchAll() /home/flyfish/www/flyshop/lib/Varien/Data/Collection/Db.php:687
[0x00000000124668f0] _fetchAll() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Entity/Collection/Abstract.php:1045
[0x0000000012466288] _loadEntities() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Entity/Collection/Abstract.php:869
[0x0000000012465fb0] load() /home/flyfish/www/flyshop/app/code/core/Mage/Review/Model/Observer.php:78
[0x0000000012465d10] catalogBlockProductCollectionBeforeToHtml() /home/flyfish/www/flyshop/app/code/core/Mage/Core/Model/App.php:1303
[0x0000000012464c28] _callObserverMethod() /home/flyfish/www/flyshop/app/code/core/Mage/Core/Model/App.php:1278
[0x00000000124649e0] dispatchEvent() /home/flyfish/www/flyshop/app/Mage.php:416
[0x0000000012464290] dispatchEvent() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Block/Product/List.php:163
[0x0000000012463760] _beforeToHtml() /home/flyfish/www/flyshop/var/ait_rewrite/6bfe16ca572eea47db567910902c6209.php:864
[0x00000000124633b0] toHtml() /home/flyfish/www/flyshop/var/ait_rewrite/6bfe16ca572eea47db567910902c6209.php:584
[0x0000000012462e30] _getChildHtml() /home/flyfish/www/flyshop/var/ait_rewrite/6bfe16ca572eea47db567910902c6209.php:528
[0x0000000012462d38] getChildHtml() /home/flyfish/www/flyshop/var/cache/extendware/ewcore/overrides/Mage/Catalog/Block/Category/View/6362e7526f5dcb27e7f8b0b414b59004.php:85
[0x00000000124629f0] getProductListHtml() /home/flyfish/www/flyshop/app/code/local/Extendware/EWLayeredNav/Block/Override/Mage/Catalog/Category/View.php:20

[01-Apr-2012 14:27:55]  [pool magento] pid 21536
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000011a35010] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement/Pdo.php:228
[0x0000000011a34d80] _execute() /home/flyfish/www/flyshop/lib/Varien/Db/Statement/Pdo/Mysql.php:110
[0x0000000011a346f8] _execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement.php:300
[0x0000000011a34288] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:479
[0x0000000011a33ca0] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Pdo/Abstract.php:238
[0x0000000011a338e0] query() /home/flyfish/www/flyshop/lib/Varien/Db/Adapter/Pdo/Mysql.php:389
[0x0000000011a33700] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:825
[0x0000000011a33368] fetchOne() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Resource/Entity/Type.php:71
[0x0000000011a33238] getAdditionalAttributeTable() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Resource/Entity/Attribute.php:483
[0x0000000011a32be8] getAdditionalAttributeTable() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Resource/Entity/Attribute.php:500
[0x0000000011a32860] _afterLoad() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Resource/Entity/Attribute.php:108
[0x0000000011a32330] loadByCode() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Entity/Attribute/Abstract.php:118
[0x0000000011a31350] loadByCode() /home/flyfish/www/flyshop/app/code/core/Mage/Eav/Model/Config.php:423
[0x0000000011a30ce8] getAttribute() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Helper/Output.php:156
[0x0000000011a30208] categoryAttribute() /home/flyfish/www/flyshop/app/design/frontend/base/default/template/catalog/category/view.phtml:47
[0x0000000011a2fa60] +++ dump failed

[01-Apr-2012 14:27:56]  [pool magento] pid 21530
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000011a35b10] updateParamDefaults() /home/flyfish/www/flyshop/var/ait_rewrite/78778b0d1ad4bf93e846365bd2fbf33f.php:276
[0x0000000011a35750] updateParamDefaults() /home/flyfish/www/flyshop/var/ait_rewrite/78778b0d1ad4bf93e846365bd2fbf33f.php:326
[0x0000000011a351f0] getSkinBaseUrl() /home/flyfish/www/flyshop/var/ait_rewrite/78778b0d1ad4bf93e846365bd2fbf33f.php:482
[0x0000000011a350a8] getSkinUrl() /home/flyfish/www/flyshop/var/ait_rewrite/6bfe16ca572eea47db567910902c6209.php:981
[0x0000000011a32468] getSkinUrl() /home/flyfish/www/flyshop/app/code/local/Extendware/EWMinify/Block/Override/Mage/Page/Html/Head.php:126
[0x0000000011a30ca8] getCssJsHtml() /home/flyfish/www/flyshop/app/code/local/Extendware/EWCore/Block/Override/Mage/Page/Html/Head.php:55
[0x0000000011a30978] getCssJsHtml() /home/flyfish/www/flyshop/app/code/local/MageWorx/SeoSuite/Block/Page/Html/Head.php:41
[0x0000000011a2fd10] getCssJsHtml() /home/flyfish/www/flyshop/app/design/frontend/moxy/default/template/rokmagemodules/rokmage-modalheader/rokmage-head.phtml:26
[0x0000000011a2f568] +++ dump failed

[01-Apr-2012 14:28:28]  [pool magento] pid 21527
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000010c7bba0] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement/Pdo.php:228
[0x0000000010c7b910] _execute() /home/flyfish/www/flyshop/lib/Varien/Db/Statement/Pdo/Mysql.php:110
[0x0000000010c7b288] _execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement.php:300
[0x0000000010c7ae18] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:479
[0x0000000010c7a830] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Pdo/Abstract.php:238
[0x0000000010c7a470] query() /home/flyfish/www/flyshop/lib/Varien/Db/Adapter/Pdo/Mysql.php:389
[0x0000000010c7a168] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:808
[0x0000000010c79558] fetchPairs() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Product/Collection.php:840
[0x0000000010c79240] addCountToCategories() /home/flyfish/www/flyshop/app/code/community/Mage/Catalog/Block/Navigation.php:133
[0x0000000010c71d48] getCurrentChildCategories() /home/flyfish/www/flyshop/app/design/frontend/base/default/template/rokmagemodules/rokmage-magemenus/rokmage-magemenu-left.phtml:139
[0x0000000010c715a0] +++ dump failed

[01-Apr-2012 14:28:28]  [pool magento] pid 21577
script_filename = /home/flyfish/www/flyshop/index.php
[0x0000000011a3a8d8] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement/Pdo.php:228
[0x0000000011a3a648] _execute() /home/flyfish/www/flyshop/lib/Varien/Db/Statement/Pdo/Mysql.php:110
[0x0000000011a39fc0] _execute() /home/flyfish/www/flyshop/lib/Zend/Db/Statement.php:300
[0x0000000011a39b50] execute() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:479
[0x0000000011a39568] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Pdo/Abstract.php:238
[0x0000000011a391a8] query() /home/flyfish/www/flyshop/lib/Varien/Db/Adapter/Pdo/Mysql.php:389
[0x0000000011a38f40] query() /home/flyfish/www/flyshop/lib/Zend/Db/Adapter/Abstract.php:734
[0x0000000011a37cc0] fetchAll() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Category/Flat.php:276
[0x0000000011a37b20] _loadNodes() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Category/Flat.php:1229
[0x0000000011a379a0] getChildrenCategories() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Category.php:841
[0x0000000011a37690] getChildrenCategories() /home/flyfish/www/flyshop/app/code/community/Mage/Catalog/Block/Navigation.php:130
[0x0000000011a30198] getCurrentChildCategories() /home/flyfish/www/flyshop/app/design/frontend/base/default/template/rokmagemodules/rokmage-magemenus/rokmage-magemenu-left.phtml:139
[0x0000000011a2f9f0] +++ dump failed

[01-Apr-2012 14:28:48]  [pool magento] pid 21629
script_filename = /home/flyfish/www/flyshop/index.php
[0x00002ac987e2cb48] _loadPrices() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Product/Type/Configurable/Attribute/Collection.php:252
[0x00002ac987e2c660] _loadPrices() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Resource/Product/Type/Configurable/Attribute/Collection.php:132
[0x00002ac987e2bf20] _afterLoad() /home/flyfish/www/flyshop/lib/Varien/Data/Collection/Db.php:536
[0x00002ac987e2ba28] load() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:253
[0x00002ac987e2aee8] getConfigurableAttributes() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:330
[0x00002ac987e2ab10] getUsedProducts() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product/Type/Configurable.php:458
[0x00002ac987e2a990] isSalable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1264
[0x00002ac987e2a618] isAvailable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1244
[0x00002ac987e2a528] isSalable() /home/flyfish/www/flyshop/app/code/core/Mage/Catalog/Model/Product.php:1308
[0x00002ac987e28f18] isSaleable() /home/flyfish/www/flyshop/app/design/frontend/moxy/default/template/rokmagemodules/rokmage-categoryview/rokmage-categoryview.phtml:122
[0x00002ac987e28770] +++ dump failed
___________________________________________

A snippet of the Latest php-fpm error log:
[01-Apr-2012 14:26:12] WARNING: [pool magento] child 21537, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.265105 sec), logging
[01-Apr-2012 14:26:12] ERROR: failed to ptrace(PEEKDATA) pid 21537: Input/output error (5)
[01-Apr-2012 14:26:44] WARNING: [pool magento] child 21531, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.268434 sec), logging
[01-Apr-2012 14:26:44] ERROR: failed to ptrace(PEEKDATA) pid 21531: Input/output error (5)
[01-Apr-2012 14:27:01] WARNING: [pool magento] child 21528, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (6.656633 sec), logging
[01-Apr-2012 14:27:01] ERROR: failed to ptrace(PEEKDATA) pid 21528: Input/output error (5)
[01-Apr-2012 14:27:04] WARNING: [pool magento] child 21529, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.211136 sec), logging
[01-Apr-2012 14:27:55] WARNING: [pool magento] child 21536, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.207001 sec), logging
[01-Apr-2012 14:27:55] ERROR: failed to ptrace(PEEKDATA) pid 21536: Input/output error (5)
[01-Apr-2012 14:27:56] WARNING: [pool magento] child 21530, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.503186 sec), logging
[01-Apr-2012 14:27:56] ERROR: failed to ptrace(PEEKDATA) pid 21530: Input/output error (5)
[01-Apr-2012 14:28:28] WARNING: [pool magento] child 21577, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.722625 sec), logging
[01-Apr-2012 14:28:28] WARNING: [pool magento] child 21527, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.122326 sec), logging
[01-Apr-2012 14:28:28] ERROR: failed to ptrace(PEEKDATA) pid 21527: Input/output error (5)
[01-Apr-2012 14:28:28] ERROR: failed to ptrace(PEEKDATA) pid 21577: Input/output error (5)
[01-Apr-2012 14:28:48] WARNING: [pool magento] child 21629, script '/home/flyfish/www/flyshop/index.php' (request: "GET /flyshop/index.php") executing too slow (5.446961 sec), logging
[01-Apr-2012 14:28:48] ERROR: failed to ptrace(PEEKDATA) pid 21629: Input/output error (5)
_____________________________________________

I also noticed that the server is not using much memory:

Mem:  16777216k total,  1204040k used, 15573176k free

My.conf settings:

query_cache_size = 128M
innodb_buffer_pool_size = 512M
open-files-limit = 8192
table_cache=4096

I just noticed that someone changed my innodb_buffer_pool_size to 512M. Shouldn't this be set to 80% of available ram? So I have 16gb ram so it should be set at 12G; however, I set it at 10G. What do you think? I made that change and restart everything. Php-fpm is still spiking cpu. Here is just 1 php-fpm process:

23942 user   17   0  507m  99m  27m R   90.9%CPU   0.6   0:03.46 php-fpm

I'm sure there may be more information you will need to help, so just let me know what you guys need to help me figure this out. Thank you.

Can you show statistics of APC? (using `apc.php`). And I would suggest to install monitoring system like `munin` with additional `pymunin` plugins to monitor nginx, apc, fpm and memcached. Need to see how CPU load correlates with network load. — Dada, Apr 01 '12 at 22:35
Thanks. I'll look into installing munin and pymunin. Never heard of either. You may have to look fast at the apc settings from above link because I'm going to have to stop php-fpm before my host shuts down my server again. — Terry Dunford, Apr 01 '12 at 22:41
I believe that I already tried disabling APC and the problem still existed. — Terry Dunford, Apr 01 '12 at 22:42
what is the configuration of php? (output of `phpinfo();`). It seems that your `joomla` working ok, only `magento` is giving troubles.. — Dada, Apr 01 '12 at 22:45
apc bcmath bz2 calendar Core ctype curl date dom ereg exif fileinfo filter ftp gd gettext gmp hash iconv imap ionCube Loader json libxml mbstring mcrypt memcache mhash mysql mysqli openssl pcntl pcre PDO pdo_mysql pdo_sqlite Phar readline Reflection session shmop SimpleXML soap sockets SPL standard tokenizer wddx xml xmlreader xmlrpc xmlwriter xsl zip zlib [Zend Modules] the ionCube PHP Loader — Terry Dunford, Apr 01 '12 at 22:47
I don't know how to locate the problem process and why the php-fpm processes are spiking the cpu. I will look into installing the munin and pymunin now if that's what is going to help me to do that. — Terry Dunford, Apr 01 '12 at 22:51
What does this error mean: ERROR: failed to ptrace(PEEKDATA) pid 21629: Input/output error (5) ...from php-fpm error log. — Terry Dunford, Apr 01 '12 at 22:53
check that you have it enabled in Magento: System->Cache Management enabled for all cache types and in System->Tools->Compilation click “Run Compilation Process” button. — Dada, Apr 01 '12 at 22:55
For MySQL tuning there is a pretty nifty script that can give you suggestions on where you should focus your optimization - mysqlautotuner. — pauska, Apr 01 '12 at 23:03
What does compilation have to do with anything? I turned compilation off and probably won't turn it on until after I'm finished with my store. I have been messing around with this problem way too long. I really need to find someone who knows what their doing. Anyone one of you need some extra money to resolve this problem for me today? But, please, only experienced takers. I already wasted $500 on Friday and got absolutely no help from it. — Terry Dunford, Apr 01 '12 at 23:08
I don't usually pimp my employer directly, but we do a lot of [specialist Magento hosting](http://www.anchor.com.au/supported-applications/magento-web-hosting/), specifically high-capacity sites. We haven't yet met a Magento store we couldn't tame. — womble, Apr 02 '12 at 05:27

score 1 · Answer 1 · answered Apr 01 '12 at 23:39

1

Try to adjust settings in your magento/app/etc/local.xml

<config>
    <global>
        <session_save><![CDATA[]]></session_save>
        <session_save_path><![CDATA[tcp://127.0.0.1:11211?persistent=1&weight=2&timeout=10&retry_interval=10]]></session_save_path>
        <session_cache_limiter><![CDATA[]]></session_cache_limiter>
        <cache>
            <backend>memcached</backend>
            <memcached>
                <servers>
                    <server>
                        <host><![CDATA[127.0.0.1]]></host>
                        <port><![CDATA[11211]]></port>
                        <persistent><![CDATA[1]]></persistent>
                        <weight><![CDATA[]]></weight>
                        <timeout><![CDATA[]]></timeout>
                        <retry_interval><![CDATA[]]></retry_interval>
                        <status><![CDATA[]]></status>
                    </server>
                </servers>
                <compression><![CDATA[0]]></compression>
                <cache_dir><![CDATA[]]></cache_dir>
                <hashed_directory_level><![CDATA[]]></hashed_directory_level>
                <hashed_directory_umask><![CDATA[]]></hashed_directory_umask>
                <file_name_prefix><![CDATA[]]></file_name_prefix>
            </memcached>
        </cache>
        <remote_addr_headers>
            <header1>HTTP_X_REAL_IP</header1>
            <header2>HTTP_X_FORWARDED_FOR</header2>
        </remote_addr_headers>
    </global>
</config>

answered Apr 01 '12 at 23:39

Dada

634
5
14

Ok, I've added the code successfully and I'm keeping an eye on the php-fpm processes. I think I may have discovered what the problem was. I had this listed in local. xml: <![CDATA[files]]>, so you think that could've been the problem? – Terry Dunford Apr 02 '12 at 00:24
Problem still remains. Any other suggestions anyone? – Terry Dunford Apr 02 '12 at 01:04
I can only guess that there is a problem with I/O somewhere. It can be either server-side: buggy php, some php module, mysql client library or server, memcached or software-side: magento settings. If you run `php5-cgi -i` - how fast (subjectively) you get an output? Can you get stats from memcached `echo stats | nc 127.0.0.1 11211`? It seems that there is no direct answer to your question but rather you need some hand-on tuning of your site. – Dada Apr 02 '12 at 08:59
Posted memcached stats above. – Terry Dunford Apr 02 '12 at 20:19
Just want to confirm: are these proper steps to install munin on nginx: https://github.com/jnstq/munin-nginx-ubuntu ...and will it be alright to recompile? Is there anything that I should be aware of before doing it? – Terry Dunford Apr 02 '12 at 20:26
Is php-fpm really even necessary if I'm using APC, Memcached and a Full Page Cache for magento? My joomla is lightening fast and I had gotten my magento store to load in 1.5 seconds on a product list page that had over 300 products. Also, you mentioned php5-fpm, is there a big difference between that and php-fpm which is what I'm using? – Terry Dunford Apr 02 '12 at 20:57
`php-fpm` is a connection manager for php - without it joomla, magento, APC and other stuff would not work at all. On Debian system php-fpm is called php5-fpm, but it is the same as on Centos. Your `memcached` stats suggests that it is working. Also we know that `APC` was working before. I can further suggest that the problem can be with mysql, some other conflicting php module or disk/swap operation (munin would be handy to eliminate that one). Try to run `php-cgi -i` to see how fast the output. – Dada Apr 02 '12 at 21:45
For `munin` you can use the guide [here](http://toastergremlin.com/?p=264) Skip the steps related to `apache`. Instead add to you default server nginx config: `location /munin { alias /var/cache/munin/www; }`, reload `nginx` config and see if it is working. That conversation here becomes difficult to follow and too localized, I suggest you contact me or somebody other by mail (you don't have enough reputation to chat here). My address is in profile. – Dada Apr 02 '12 at 21:55
Output of php-cgi -i took about a second. When my host is done and if he hasn't solved the problem, then I will try to install munin again. Thanks for your help. – Terry Dunford Apr 02 '12 at 22:00

score 1 · Answer 2 · answered Jun 27 '12 at 15:01

1

You have this:

php_admin_value[memory_limit] = 128M

Magento's current System Requirements are "Memory_limit no less than 256Mb (preferably 512)".

answered Jun 27 '12 at 15:01

Terje Sten Bjerkseth

111
2

score 0 · Answer 3 · answered Apr 02 '12 at 05:29

0

Dis you install munin yet?

What do you mean by "consuming the entire network"? Are you implying that you have that much traffic on one magento server?

How many php req/s do you have in averege and peak?

Dis you run http://mysqltuner.pl?

answered Apr 02 '12 at 05:29

3molo

4,330
5
32
46

The CPU is maxing out and affecting the other VPS servers on the network. I do not currently have my own dedicated VPS, though that will change in 2 months. I have to get to get by until then. My host shut down my server and is diagnosing the problem right now. Yes, I did install and run mysqltuner a couple months ago and last night I installed mysqlreport and the report looks fine. I personally think that the issue is with memcached. Here are the memcached stats: – Terry Dunford Apr 02 '12 at 19:49
accepting_conns 1 bytes 9539801 bytes_read 15175135 bytes_written 127526956 cmd_flush 3 cmd_get 43854 cmd_set 2137 connection_structures 19 curr_connections 14 curr_items 1029 delete_hits 60 delete_misses 26 expired_unfetched 87 get_hits 20608 get_misses 23246 hash_bytes 524288 hash_is_expanding 0 hash_power_level 16 incr_hits 71 incr_misses 9 libevent 1.4.13-stable limit_maxbytes 1073741824 pid 11436 pointer_size 64 reclaimed 599 reserved_fds 20 rusage_system 2.823570 rusage_user 1.270806 threads 4 time 1333382983 total_connections 76 total_items 2137 touch_hits 0 touch_misses 0 uptime 45835 – Terry Dunford Apr 02 '12 at 19:52
I did install munin, but was having a hard time accessing site.com/munin probably because I needed to add nginx rules, right? How do I get it to work? – Terry Dunford Apr 02 '12 at 19:54
One "professional" told me that the best thing would be to get a dedicated VPS or a VPS with "dedicated CPU time", as he put it. I've tried everything that I can think of to narrow down the problem. Tried profiling, checked all the config settings of nginx, php-fpm, memcached, apc, ran mysql reports, but nothing has worked. Hopefully my host will resolve the problem, as he is currently working on it. If not, then I hope you can continue to help me. Not sure about php requests – Terry Dunford Apr 02 '12 at 19:59
...but I will find out when my server comes back online and post the results for you. Hopefully my host will fix the problem so I can be done with this. – Terry Dunford Apr 02 '12 at 20:01
Terry, please update your post with relevant information. I would like to know how many magento hits (not static content (js|css|png|etc)) you have, as I know magento is one heavy beast and it might very well be that you have too much traffic for your server to handle, no matter if a VPS or not. – 3molo Apr 03 '12 at 06:42
The Magento site is still in development; however, we've replaced php-fpm with php-fastcgi and replaced APC with e-accelerator. So far so goood. I also was able to successfully install munin. It looks like installing php-fastcgi and e-accelerator has fixed the problem, but time will tell. – Terry Dunford Apr 03 '12 at 21:37

Magento, NGINX, PHP-FPM, APC, MEMCACHED, 16gb Ram CentOS, Spiking PHP-FPM to 100% CPU

3 Answers3