Batch proccesing of long execution time script

Question

I searched around but i couldn't find a solution other than set_time_limit(0) which won't work on most of the shared hosting around.

Basically i have a script that send messages to my user's friends when they want. Some of my users have +4000 friends and the script gets into trouble.

Currently im calling this script in the background with AJAX. As i don't need/want the user to wait until this finish i would love to have some kind of background proccesing.

My current code:

global $client, $emails, $subject, $message;
_info("got on_auth_success cb, jid ".$client->full_jid->to_string());
$client->set_status("available!", "dnd", 10);

set_time_limit(60*10);

if( count($emails) < 40 ){

    foreach( $emails as $email )
    {
        $msg = new XMPPMsg(array('to'=>'-'.$email.'@chat.facebook.com'), $message);
        $client->send($msg);  
        sleep(1);
    }

}
else
{
    $counter = 0;
    //Lets create batches
    foreach( $emails as $email )
    {
        $counter++;
        $msg = new XMPPMsg(array('to'=>'-'.$email.'@chat.facebook.com'), $message);
        $client->send($msg);
        sleep(1);

        if( $counter == 50 )
        {
            sleep(10);
            $counter = 0;
        }
    }
}
$client->send_end_stream();

Would be a good solution to use exec ? like for example

exec("doTask.php $arg1 $arg2 $arg3 >/dev/null 2>&1 &");

I need a solution that works on most of the hosting as this is a wordpress plugin that can be installed on any host. Thanks!

You should _never_ do such tasks in a synchronous way! Instead either delegate such jobs to some soft of cron system, so that messages get sent out when time permits. Alternatively use a "poor mans cron" if not even a cron system is available. But you certainly should store the jobs and process them asynchronously. — arkascha, Jul 13 '13 at 16:21
No sorry, I have no code examples around. I implemented a complex system doing such tasks some 10 years ago, such tasks can get _really_ complex. But for the basic idea there are hundreds of examples on google. The basic approach: you save all "jobs to be done" in the database and then process them one by one whenever a task can get triggered: by a cron job or by a poor man's cron, just as mentioned before. Then it is just a queston of tuning how many jobs are processed in one go. — arkascha, Jul 13 '13 at 21:54

score 0 · Accepted Answer · edited May 23 '17 at 12:11

It would be ideal if you put this into some sort of cron. Build a list of emails to send and store them in a queue and have a cron script of some sort process that queue. Wordpress does have it's own cron mechanism (see wp_cron), but it might be difficult on low-traffic sites to run frequently enough to send that number of emails. Granted, you could use cron proper to make sure wp_cron is run. You should see How to execute a large PHP Script? as it's very related.

If you really want it to be synchronous, since you are using background ajax you could also just make a number of smaller ajax calls. For example, you could make your ajax script perform 20 emails at a time, and then return the number of emails remaining. Then your client code on the browser knows there's still some left and calls the background php via ajax again, perhaps with a counter indicating the current position. [Edit: But this is reliant on your users being patient enough to keep their browsers open to complete the send, and they may well need to have relatively stable connection to the internet.]

I really don't need the user to wait until the proccess is complete. I could just run it in the background. I saw some plugins use a sort of self ping in order to have the proccess running on the background. — chifliiiii, Jul 13 '13 at 19:12
Ah, yes that is another way to accomplish the background processing. — EPB, Jul 13 '13 at 19:16

Batch proccesing of long execution time script

1 Answers1