PHP: Processing thousands of entries, script dies after certain amount

Question

I'm calling an MLS service that responds with 4000+ records ... and I need to process each and every one of them, as well as insert all of the meta data per listing.

I'm able to get to about 135 (* 150 meta records) and then the script apparently stops responding, or at least stops processing the rest of the data.

I've added the following to my .htaccess file:

php_value memory_limit 128M

But this doesn't seem to help me any. Do I need to process chunks of the data at a time, or is there another way to ensure that the script will indeed finalize?

If it is the time limit, make sure your tables are indexed. Doing 150 queries probably shouldn't be enough to run into the time limit unless it's set really low. — Mike, Dec 19 '12 at 01:25
We need to see your code to determine whats actually happening. — Shiplu Mokaddim, Dec 19 '12 at 01:59

score 3 · Answer 1 · answered Dec 19 '12 at 01:21

You should probably enable display_errors and error_reporting to get a better analysis of why the script isn't processing.

However, you should also consider making sure the time limit isn't being hit by calling:

set_time_limit( 0 );

This will give you an unlimited time period. You can also just set it to something relatively high, like 600 (10 minutes)

score 1 · Accepted Answer · answered Dec 19 '12 at 01:22

1

It isn't the memory- it's most likely the script execution time.

Try adding this to your htaccess, then restart apache:

php_value max_execution_time 259200

answered Dec 19 '12 at 01:22

FredTheWebGuy

2,546
3
27
34

PHP: Processing thousands of entries, script dies after certain amount

2 Answers2