Too slow Http Client in Zend Framework 1.12

Question

I want to send ~50 requests to different pages on the same domain and then, I'm using DOM object to gain urls to articles.

The problem is that this number of requests takes over 30 sec.

for ($i = 1; $i < 51; $i++)
{
    $url = 'http://example.com/page/'.$i.'/';             

    $client = new Zend_Http_Client($url);
    $response = $client->request();
    $dom = new Zend_Dom_Query($response); // without this two lines, execution is also too long
    $results = $dom->query('li');         //
}

Is there any way to speed this up?

score 1 · Answer 1 · answered Jul 03 '13 at 12:37

1

I can't think of a way to speed it up but you can increase the timeout limit in PHP if that is your concern:

for($i=1; $i<51; $i++) {
  set_time_limit(30);    //This restarts the timer to 30 seconds starting now
  //Do long things here
}

answered Jul 03 '13 at 12:37

Joel Lord

2,175
1
18
25

simplyray · Accepted Answer · 2013-07-04T09:49:45.420

It's a generel problem by design - not the code itself. If you're doing a for-loop over 50 items each opening an request to an remote uri, things get pretty slow since every requests waits until responde from the remote uri. e.g.: a request takes ~0,6 sec to been completed, multiple this by 50 and you get an exection time of 30 seconds!

Other problem is that most webserver limits its (open) connections per client to an specific amount. So even if you're able to do 50 requests simultaneously (which you're currently not), things won't speed up measurably.

In my option there is only one solution (without any deep going changes): Change the amout of requests per exection. Make chunks from e.g. only 5 - 10 per (script)-call and trigger them by an external call (e.g. run them by cron).

Todo: Build a wrapper function which is able to save the state of its current run ("i did request 1 - 10 at my last run, so now I have to call 11 - 20) into a file or database and trigger this function by an cron.

Example Code (untested) for better declaration;

[...]

private static $_chunks = 10; //amout of calls per run

public function cronAction() {

    $lastrun = //here get last run parameter saved from local file or database

    $this->crawl($lastrun);

}

private function crawl($lastrun) {

    $limit = $this->_chunks + $lastrun;

    for ($i = $lastrun; $i < limit; $i++)
    {
        [...] //do stuff here
    }

    //here set $lastrun parameter to new value inside local file / database

}

[...]

Too slow Http Client in Zend Framework 1.12

2 Answers2