3

I have the following set-up:

  • Zend Framework 1.10.8
  • database adapter: pdo_mysql
  • memory limit: 128MB
  • table type: myisam
  • table size: 4 MB
  • number of records: 22.000
  • number of columns: aprox. 70

I want to select everything from the table, which should be no problem at all:

$table = new Application_Model_DbTable_Foo();
$everything = $table->fetchAll();

Doing this I run into a "memory exhausted" problem.

I wrote a script using PDO_MYSQL without Zend Framework and the memory usage was fine and it worked all right.

So I must be missing something here. Any hint highly appreciated. :-)

Charles
  • 50,943
  • 13
  • 104
  • 142
herrjeh42
  • 2,782
  • 4
  • 35
  • 47

4 Answers4

2

22k rows alone can a significant amount of data, depending on what your script is doing. If increasing the memory limit isn't an option, you might want to try this. Grab 'chunks' of rows. You can do this with a LIMIT. Set your page size to something that fits in memory and then process that. Keep increasing your OFFSET until you've processed all your data.

Chris Henry
  • 11,914
  • 3
  • 30
  • 31
2

Thanx, everybody, for having a look at my problem. :-)

For further reference, if someone else has this problem, I ended up like this:

 $db = Zend_Db_Table_Abstract::getDefaultAdapter();
   $result = $db->query("SELECT * from footable");
   foreach ($result as $row) {            
           //print $row["field"]; 
  }
  unset($result);

... still considering to switch to the "get it in chunks using LIMIT x,y"-suggestion.

herrjeh42
  • 2,782
  • 4
  • 35
  • 47
1

I worked on the Zend_Db component quite a bit when I was on the Zend Framework project (up until ZF 1.0).

As @netcoder says, the Zend_Db_Table component creates objects for each row fetched. Objects take more memory than primitive arrays, but they also give you the opportunity to use the OO usage and save() etc.

The real trouble is that you're doing fetchAll() on a large data set. Even if your memory limit could handle 22000 rows, tomorrow you will need 44000 rows, and next month 100000 rows.

Regardless of the Zend_Db implementation, or if you use PDO with no framework, you need to learn to manage large data sets without loading them completely into memory, because that will inevitably lead to memory exhaustion. This is true for any library or framework or even any language.

You can try using LIMIT as @Chris Henry suggests, so you are sure to fetch a fixed quantity of rows that is not too much.

Alternatively, you can fetch the data a row at a time so you only have one row in memory. The Zend_Db_Table class does not provide a means to do this; you'll have to run SQL queries yourself with the Zend_Db_Adapter interface and then loop over the results with fetch().

See examples at http://framework.zend.com/manual/en/zend.db.statement.html

Bill Karwin
  • 538,548
  • 86
  • 673
  • 828
0

Zend_Db uses a lot of abstraction, classes, etc. For each row in your result set, it creates an object (Zend_Db_Table_Row). The object for each row uses way more memory than an array with key-value pairs would.

netcoder
  • 66,435
  • 19
  • 125
  • 142