0

We have a website that has 'projects' on it with a given number of people subscribed to each project.

At the 'end' of a project, all the subscribers are gathered and processed. In this case, there are roughly 1,000 subscribers whose data needs to be pulled and processed. There is also some related data that is stored with each data set in an array.

The last time we processed a large order, ~300 items, PHP ran out of memory. We boosted the memory and it was able to process it. I do not think that would be the case this time.

We currently use jobs that pull the project to process the subscribers. In this job, an SQL query is executed to pull all the 'subscribers' and store them and their related data in the array. The array is then iterated over to create individual jobs to process the individual subscribers.

My question is:

Is there a way to do this in 'blocks' or something? Or ANY better way that will lessen the impact on memory? I want this to be scalable in the event we have thousands of subscribers.

Right now this is the flow:

  • Project 'ends'
  • Job is kicked off that sets some flags, and pulls all of the subscribers
  • The array of data from MySQL (the subscribers) is looped over, and an individual job for each subscriber is created.
  • Each subscriber job is processed by the engine.

I am just having hard time pinpointing the best process to do this.

valentinas
  • 4,277
  • 1
  • 20
  • 27
Barry Chapman
  • 6,690
  • 3
  • 36
  • 64
  • It is NOT time critical. – Barry Chapman Mar 20 '13 at 23:01
  • 1
    You need to divide the users and do it in parts. Just write a script, that handles smaller amount of users, add it to the cron and schedule it for like every hour at night (or sooner if it doesn't take that long). The script writes in the database, that the certain user was handled already, so the next query (after an hour) will handle the next 'packet' of users. – sobol6803 Mar 20 '13 at 23:13

1 Answers1

0

I'd use the main RS sorted on prefeerably int-key, and only process one subscriber.

At end of partial job, save the id of the job done. At final echo back if your done or not, and have a parent script calling it:

Add an usleep at end of processscript to let other players in..

 while(file_get_contents('http://yourscript.php') != false);

(not using include adds overhead but saves you from memory leaks)

Teson
  • 6,644
  • 8
  • 46
  • 69