cURL Multi Threading with PHP

Question

I'm using cURL to get some rank data for over 20,000 domain names that I've got stored in a database.

The code I'm using is http://semlabs.co.uk/journal/object-oriented-curl-class-with-multi-threading.

The array $competeRequests is 20,000 request to compete.com api for website ranks.

This is an example request: http://apps.compete.com/sites/stackoverflow.com/trended/rank/?apikey=xxxx&start_date=201207&end_date=201208&jsonp=";

Since there are 20,000 of these requests I want to break them up into chunks so I'm using the following code to accomplish that:

foreach(array_chunk($competeRequests, 1000) as $requests) {
    foreach($requests as $request) {
        $curl->addSession( $request, $opts );
    }

}

This works great for sending the requests in batches of 1,000 however the script takes too long to execute. I've increased the max_execution_time to over 10 minutes.

Is there a way to send 1,000 requests from my array then parse the results then output a status update then continue with the next 1,000 until the array is empty? As of now the screen just stays white the entire time the script is executing which can be over 10 minutes.

do you have to run this script through your server, can't you just run it manually? ... 20,000 requests is a lot of requests, this will most likely have to run in the background — hackattack, Sep 12 '12 at 18:20
I did want to run it as cron job eventually..for now I'm executing it in browser.. — user1647347, Sep 12 '12 at 18:25
I'm already using curl multi... "the code im using is http://semlabs.co.uk/journal/object-oriented-curl-class-with-multi-threading. " — user1647347, Sep 12 '12 at 18:51

score 16 · Answer 1 · edited May 28 '20 at 08:26

16

The above accepted answer is outdated, So, correct answer has to be upvoted.

http://php.net/manual/en/function.curl-multi-init.php

Now, PHP supports fetching multiple URLs at the same time.

edited May 28 '20 at 08:26

jsims281

2,206
2
30
57

answered May 19 '16 at 12:10

Mani

440
4
15

5

The correct answer is the one I gave at the time. There is no need to correct old questions on SO just to keep up with science and development. At the time, that was the answer. You don't know if someone uses old PHP versions or not, it's possible they are. Don't assume everyone runs the latest. If you claim a certain support exists in PHP, you should also mention what version YOU are talking about. – Glenn Plas Jul 19 '16 at 16:58

score 8 · Accepted Answer · answered Oct 03 '12 at 18:44

8

This one always does the job for me... https://github.com/petewarden/ParallelCurl

answered Oct 03 '12 at 18:44

Glenn Plas

1,608
15
17

1

An important part of the ParallelCurl code is the bugfix for curl_multi_select (https://bugs.php.net/bug.php?id=63411) for PHP 5.3.18+ starting on line 118 in parallelcurl.php, which probably was the fix the OP was looking for (and anyone else with broken curl_multi scripts) – zoltar Apr 27 '14 at 23:16
1

This answer is outdated and the correct answer is http://php.net/manual/en/function.curl-multi-init.php – Mani May 19 '16 at 12:09

score 5 · Answer 3 · edited Dec 31 '13 at 08:43

5

https://github.com/krakjoe/pthreads

enter image description here

You may thread in PHP, the code depicted is just horrible thread programming, and I don't advise that is how you do it, but wanted to show you the overhead of 20,000 threads ... it's 18 seconds, on my current hardware which is a Intel G620 ( dual core ) with 8gigs of ram, on server hardware you can expect much faster results ... how you thread such a task is dependant on your resources, and the resources of the service you are requesting ...

edited Dec 31 '13 at 08:43

Mohsen Safari

6,669
5
42
58

answered Sep 13 '12 at 18:55

Joe Watkins

17,032
5
41
62

the "programming" itself is fine - its the complete lack of *thread scheduling / thread execution queue* that makes this script inefficient. In general, a good scheduler should never allow more than `insert amount of CPU cores here` to run simultaneously, except for machines with HT and threads which have extreme `wait` times and it should be able to **queue** at least 2^32 threads – specializt May 19 '16 at 12:23

Nelson · Answer 4 · 2012-09-12T19:02:16.590

2

Put this at the top of your php script:

set_time_limit(0);
@apache_setenv('no-gzip', 1);//comment this out if you use nginx instead of apache
@ini_set('zlib.output_compression', 0);
@ini_set('implicit_flush', 1);
for ($i = 0; $i < ob_get_level(); $i++) { ob_end_flush(); }
ob_implicit_flush(1);

that would disable all caching the web server or php may be doing, making your output be displayed on the browser while the script is running.

Pay attention to comment out the apache_setenv line if you use nginx web server instead of apache.

Update for nginx:

So OP is using nginx, that makes things a bit trickier as nginx doesn't let to disable gzip compresion from PHP. I also use nginx and I just found out I have it active by default, see:

cat /etc/nginx/nginx.conf | grep gzip
    gzip on;
    gzip_disable "msie6";
    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

so you need to disable gzip on nginx.conf and restart nginx:

/etc/init.d/nginx restart

or you can play with the gzip_disable or gzip_types options, to conditionally disable gzip for some browsers or for some page content-types respectively.

edited Sep 12 '12 at 19:02

answered Sep 12 '12 at 18:27

Nelson

49,283
8
68
81

commented that line out..trying it now but it seems that it's still not outputting any results until the script is done executing each all domains instead of each batch – user1647347 Sep 12 '12 at 18:44
yeah just confirmed that it doesn't output anything until the very end after the script has completed all 20,000 – user1647347 Sep 12 '12 at 18:58
do you have any ideas for better more efficient processing? Basically my database has 20,000 domains and queries compete.com using their api to get the rank and then writes the ranks to the database. Ultimately I want to schedule the script to run each month automatically so I was hoping to avoid JavaScript or any other client side scripting. – user1647347 Sep 12 '12 at 19:04
See my **nginx update** above to see if disabling gzip makes it work for your case. – Nelson Sep 12 '12 at 19:05
To anyone, we need to make ajax calls from js to resolve that^ Disabling the caching is a bad idea, say someone open a few thousands http requests, and the server will crash down, as he is going to swap to the limit. – NVRM Mar 17 '18 at 21:42

cURL Multi Threading with PHP

4 Answers4

Linked