run PHP script from cli : php.ini memory_size not taken into account

Question

I am trying to run a php script (magento reindexer script) from the command line. The script consumes a lot of memory so I am getting the following error: PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /home/karanta/www/karanta.fr/lib/Zend/Db/Adapter/Abstract.php on line 691

To fix this issue, I edited the /etc/php5/cli/php.ini file and set memory_limit = 2048M.

To check the configuration, I run a script containing phpinfo(); from the cli and I see: memory_limit => 2048M => 2048M, so it seems the configuration is taken into account properly. ini_get('memory_limit); also returns 2048M.

However when I rerun the reindex script, I still get PHP Fatal error: Allowed memory size of 536870912 bytes exhausted as if the memory_limit was still 512M. The script is unable to complete, I am out of ideas how to augment the memory_limit to allow the script to complete.

Edit: I also tried adding the instruction ini_set("memory_limit", -1); directly into the PHP script and it still hangs with the same PHP Fatal error: Allowed memory size of 536870912 bytes exhausted.

Extra information:

Server is a dedicated machine at ovh running Debian GNU/Linux 7.5 with 64GB RAM!

php -v returns:

PHP 5.6.12-1 (cli) Copyright (c) 1997-2015 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies

free -k -h during script execution returns:

               total        used        free      shared  buff/cache     available
Mem:            62G        1,8G         48G         84M          12G         60G
Swap:          1,0G          0B        1,0G

ps -aux returns:

karanta  25568  5.6  0.0 229484 41824 pts/1    S    07:54   0:04 php shell/indexer.php --reindex catalog_url

Following @IgorGreg's recommandation, I tried setting the memory limit using Zend_Memory. I wrote this php script that I run from the cli in replacement of the shell/indexer.php script.

<?php
require_once 'app/Mage.php';
$app = Mage::app('admin');
umask(0);
$memoryManager = Zend_Memory::factory('none');
$memoryManager->setMemoryLimit(-1);
$process = Mage::getModel('index/process')->load(3);
$process->reindexAll();
?>

Still getting the same error.

score 7 · Accepted Answer · answered Sep 22 '15 at 06:57

7

Magento in CLI mode parses the .htaccess file and if it finds a memory limit, or any other PHP settings there, applies them (yes, seriously....)

The responsible code is in shell/abstract.php:

/**
 * Parse .htaccess file and apply php settings to shell script
 *
 */
protected function _applyPhpVariables()
{
    $htaccess = $this->_getRootPath() . '.htaccess';
    if (file_exists($htaccess)) {
        // parse htaccess file
        $data = file_get_contents($htaccess);
        $matches = array();
        preg_match_all('#^\s+?php_value\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
        if ($matches) {
            foreach ($matches as $match) {
                @ini_set($match[1], str_replace("\r", '', $match[2]));
            }
        }
        preg_match_all('#^\s+?php_flag\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
        if ($matches) {
            foreach ($matches as $match) {
                @ini_set($match[1], str_replace("\r", '', $match[2]));
            }
        }
    }
}

Solution

Do not set memory limit and execution time via .htaccess, use the php.ini or the virtual host configuration.

answered Sep 22 '15 at 06:57

Fabian Schmengler

24,155
9
79
111

Awesome, the indexer script doesn't reach the memory limit anymore! However, the reindexation still doesn't complete successfully, it now gets stuck (process has cpu at 0%, nothing happens). I already tried killing the process and relauching the reindex script, same thing happens. I suspect a big MySQL query blocking the reindexation, is that possible? Any idea how to debug that? – fkoessler Sep 22 '15 at 09:53
after killing the process you should delete the lock files in `var/locks`. To monitor MySQL you can use the `mtop` tool or just `SHOW PROCESSLIST;` – Fabian Schmengler Sep 22 '15 at 09:58
Yup, reindexing gets stuck because of queries such as `DELETE FROM 'core_url_rewrite' WHERE (url_rewrite_id IN('5498439', '5498441', '5498443', '5498445', ...` that time out. I already set `innodb_lock_wait_timeout` to `1000` and that's still not enough! Would truncating the `core_url_rewrite` table fix this issue? – fkoessler Sep 22 '15 at 13:11
1

Probably, for a while. But don't do it if you have custom URL rewrites, they will be lost. The URL index of Magento CE is a big mess, more infos here: http://magento.stackexchange.com/questions/17553/magento-core-url-rewrite-table-excessively-large – Fabian Schmengler Sep 22 '15 at 13:14

Fiasco Labs · Answer 2 · 2015-09-13T17:44:08.987

In Linux systems, there can be three separate memory limit settings, one for the Apache DSO module php instance, one for the Command Line php instance and one for the cgi php instance.

Each installation will have its own php.ini file, for example /etc/php5/apache2/php.ini and /etc/php5/cli/php.ini and may have the ability to override the set memory_limit turned off (typically found in shared hosting).

The php.ini file for cli will need to have its memory_limit increased.

Another spot of trouble can occur if you are using the apc cache. It has to be enabled by adding the following to the apc config file.

apc.enable-cli=1

A third source of trouble if you are running under cPanel/WHM is that there can be several different versions of php installed due to the security restrictions put in place. I had to contact our hosting service provider to run that one down. Try using which php from the command line where you are having the failure and make sure it's the same version that's accessible via cron scripts. The php.ini file for that particular version also has to be set.

I am editing the `/etc/php5/cli/php.ini` file as I start the php script from the command line. (also, I added some extra info at the bottom of the question) — fkoessler, Sep 14 '15 at 12:15

score 0 · Answer 3 · edited Jun 20 '20 at 09:12

0

Setting memory_limit to -1 means no memory limit. However, if you set this value in php.ini or .htaccess it may be overwritten in run-time by the script.
To be sure it'll take effect, it's better to use ini_set("memory_limit", -1); in the script.
Generally, setting memory_limit to -1 in php.ini is not a good idea. It may cause your system to hang under certain circumstances.

Magento is built on Zend framework 1 which has its own memory manager. The documentation says you may set the memory limit like so:

$memoryManager->setMemoryLimit(-1);

It doesn't need to be -1, it may be 1,073,741,824 , or any integer, in bytes.
I haven't tried it though.

edited Jun 20 '20 at 09:12

Community

1
1

answered Sep 13 '15 at 09:57

Greg Kelesidis

1,069
14
20

1

I added `ini_set("memory_limit", -1);` to my php script and I'm still getting the `PHP Fatal error: Allowed memory size of 536870912 bytes exhausted` error! How is that possible? – fkoessler Sep 13 '15 at 11:30
most probably the memory_limit setting is being overwritten somewhere after the added statement. – gyaani_guy Sep 13 '15 at 16:13
Tried it, still getting the same `PHP Fatal error: Allowed memory size` – fkoessler Sep 14 '15 at 14:02

score 0 · Answer 4 · answered Sep 22 '15 at 08:21

Please follow the following step.

1 . change the following into .htaccess to your requirement php_value memory_limit 256M => php_value memory_limit 2048M

Instead of reindexing all in one,do reindex one by one like

php indexer.php --reindex catalog_product_price

There is need to increase memory limit via .htaccess is because of you can see in abstract.php it will refer .htaccess configuration.

Let me know if you have still any query

Thanks

run PHP script from cli : php.ini memory_size not taken into account

4 Answers4

Solution