0

I have a large CSV file containing Inventory data to update (more than 35,000 rows). I created a method which extends Mage_Catalog_Model_Convert_Adapter_Productimport to do the inventory update. Then I used an Advanced Profile to do the update which calls that method.

It's working very well when I run the profile manually. The problem is when I use an extension which handles the profile running in cronjob, the system takes too long to load and parse the CSV file. I set the cronjob to run everyday at 6:15am, but the first row of the file wouldn't be processed until 1:20pm the same day, it takes 7 hours to load the file.

That makes the process stop in the middle somehow, less than 1/3 records being processed. I've been frustrating trying to figure out why, trying to solve the problem, but no luck.

Any ideas would be appreciated.

Alex Phelan
  • 1
  • 1
  • 1
    It seems very strange that it is taking "7 hours" to load the file. Have you confirmed that it is actually starting at 6:15am? My guess is that there is some [misunderstanding with the timezones](http://stackoverflow.com/questions/7580582/magento-cron-tab-job-time-zone) – nachito Jun 23 '12 at 16:56
  • 1
    I'd be logging the snot out of the method you have created. Log it out to you own log file `Mage::log('blah blah', null, 'yourlogfile.log');` so you can see what's going on – CCBlackburn Jun 24 '12 at 08:08
  • Make sure that you are processing only as many rows per iteration as your server resources allow (number_of_records). – benmarks Jun 24 '12 at 13:33

1 Answers1

0

Varien_File_Csv is the class that parses your csv file. It takes too much memory.

Function to log memory amount used and peak memory usage,

public function log($msg, $level = null)
{
if (is_null($level)) $level = Zend_Log::INFO;

$units = array('b', 'Kb', 'Mb', 'Gb', 'Tb', 'Pb');
$m = memory_get_usage();
$mem = @round($m / pow(1024, ($i = floor(log($m, 1024)))), 2).' '.$units[$i];
$mp = memory_get_peak_usage();
$memp = @round($mp / pow(1024, ($ip = floor(log($mp, 1024)))), 2).' '.$units[$ip];

$msg = sprintf('(mem %4.2f %s, %4.2f %s) ', $mem, $units[$i], $memp, $units[$ip]).$msg;

Mage::log($msg, $level, 'my_log.log', 1);
}

$MyClass->log('With every message I log the memory is closer to the sky');

You could split your csv (use same filename) and call the job multiple times. You'll need to be sure a previous call won't run same time with a newer one.

Thanks

Daniel Ifrim
  • 277
  • 3
  • 12