0

I've run into a problem where I exceed my memory limit in php 7.0.5, memory limit is set to 3072M

PHP Fatal error:  Allowed memory size of xxx bytes exhausted PDO

IIS Server running 4 G Ram total. I am connecting to a remote dedicated MySQL server.

class DB{

    private $dbh;
    private $stmt;

    public function __construct($db) {
        $host = HOST;
        $this->db = $db;
        try {
            $this->dbh = new PDO(
                "mysql:host=$host;dbname=$this->db",
                USER,
                PASS
            );          
        }catch (PDOException $e) {
            print "Error!: " . $e->getMessage() . "<br/>";
            die();
        }
    }

    public function query($query) {
        $this->stmt = $this->dbh->prepare($query);
        return $this;
    }

    public function bind($pos, $value, $type = null) {

        if( is_null($type) ) {
            switch( true ) {
                case is_int($value):
                    $type = PDO::PARAM_INT;
                    break;
                case is_bool($value):
                    $type = PDO::PARAM_BOOL;
                    break;
                case is_null($value):
                    $type = PDO::PARAM_NULL;
                    break;
                default:
                    $type = PDO::PARAM_STR;
            }
        }

        $this->stmt->bindValue($pos, $value, $type);
        return $this;
    }

    public function execute() {
        return $this->stmt->execute();
    }

    public function resultset() {
        $this->execute();
        return $this->stmt->fetchAll();
    }

    public function fetchArray() {
        $this->execute();
        return $this->stmt->fetchAll();
    }

    public function numRows() {
        $this->execute();
        return $this->stmt->rowCount();
    }

    public function single() {
        $this->execute();
        return $this->stmt->fetch();
    }

    public function insertID()
    {
        return $this->dbh->lastInsertId();
    }   
}

    function updateBooleanInRecord()
    {
        $query = "select inRecordID, recordStatus from `ems-reporting`.inrecord";
        $db = new DB("reporting");
        $db->query($query); 
        if($db->numRows() > 0){
            $out = $db->fetchArray();
            foreach($out as $output){
                if($output['recordStatus'] == "False"){
                    updateFalseRecord($output['inRecordID']);
                }
                if($output['recordStatus'] == "True"){
                    updateTrueRecord($output['inRecordID']);
                }
            }
        }
    }

    function updateFalseRecord($inRecordID){
        $query = "UPDATE `reporting`.inrecord set recordStatus = '0' where inRecordID = ?";
        $dba = new DB("ems-reporting");
        $dba->query($query);
        $dba->bind(1, $inRecordID);
        $dba->execute();
        unset($dba);
        $dba = null;
    }

    function updateTrueRecord($inRecordID){
        $query = "UPDATE `reporting`.inrecord set recordStatus = '1' where inRecordID = ?";
        $dbb = new DB("ems-reporting");
        $dbb->query($query);
        $dbb->bind(1, $inRecordID);
        $dbb->execute();
        unset($dbb);
        $dbb = null;
    }

    updateBooleanInRecord();

The dataset I am pulling from contains over 36 million rows of data and the error I am receiving appears to relate to:

public function execute() {
    return $this->stmt->execute();
}

The research I have done so far suggests that I should add the following just as after:

$this->dbh = new PDO(
                    "mysql:host=$host;dbname=$this->db",
                    USER,
                    PASS
                );

Buffered Query Attribute Addition:

$this->dbh->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, FALSE);

I have tried this, but the request runs for a few seconds, maybe 10 and then simply stops with no error being displayed in the browser or appearing in the logs. So I have zero indication of what halted the script. The memory usage does not appear to grow at all when this addition is made.

Before I added the above setAttribute I opened the task manager to watch the process consume memory until it reached its limit, then it the process fails with the following msg in the browser:

Fatal error: Allowed memory size of 3221225472 bytes exhausted (tried to allocate 1166262464 bytes) in D:\path\to\folder\index.php on line 54

Hopefully someone can point out what I may be doing wrong here, or some tips to try and locate the issue, basically any help would be greatly appreciated.


EDIT: The following SQL statements resolved my issue at the source, I wasn't thinking clearly. Thank you for the responses though, I see fetchAll() is obviously not the way to do this and leads to exactly the issue I encountered...time to sleep...

UPDATE `reporting`.inrecord set recordStatus=1 where recordStatus="True"

followed by

UPDATE `reporting`.inrecord set recordStatus=0 where recordStatus="False"
Community
  • 1
  • 1
Majickal
  • 176
  • 2
  • 16
  • 1
    Try clearing after - Instead of `foreach($out as $output){`, change it to `foreach($out as $k => &$output){` and after you `updateTrueRecord`, free up space by `$out[ $k ] = null;` – Alon Eitan May 18 '17 at 12:40
  • 1
    Even if you turn buffering OFF, if you do `fetchAll()` you are getting all rows returned into an array all in one big suck. Process rows one at a time with a `fetch()` and maybe you will complete the process – RiggsFolly May 18 '17 at 12:49
  • Also it would seem to me that processing **36 Million Row** from a web page based process is a bad approach. Nobody wants to look at 36Million rows in a table, so I woudl suggets running this as a batch process from the PHP CLI, where there are fewer limitations applied to PHP woul dbe a better idea. – RiggsFolly May 18 '17 at 12:52
  • @RiggsFolly, you are 100% correct, it is a silly way to approach it, what was I thinking!! – Majickal May 18 '17 at 13:04
  • 1
    FYI: [Your first database wrapper's childhood diseases](https://phpdelusions.net/pdo/common_mistakes) – Your Common Sense May 18 '17 at 13:19

0 Answers0