Stack Overflow
Stack Overflow

Questions tagged [curl-multi]

is refer to multiple synchronous curl transfers

197 questions
0
votes
2 answers

Sending a lot of API requests quickly with php?

I'm looking to make a bunch of REST api calls as quickly as possible. I currently have about 1,000 requests to make. The results of these calls are not needed for any processing or any thing like that. I just simply need to post all of them to the…
hcker2000
  • 603
  • 1
  • 6
  • 30
0
votes
1 answer

curl_multi is very slow to check 100 of urls

I'm using following code to check broken links in the given url. but this process is very slow. i need to speed up this process quickly. $$url_list = array( "http://goog528le.com", "http://facebook.com", "http://google.com", "http://youtube.com",…
Dumidu
  • 3
  • 2
  • 4
0
votes
1 answer

php cUrl multiple request - get all content

I'm using the following script to get the contents from multiple urls, but without any modifications it will run forever: https://raw.github.com/jmathai/php-multi-curl/master/EpiCurl.php I'm trying to get the contents of multiple urls, parallel.…
Simon
  • 5,464
  • 6
  • 49
  • 85
0
votes
0 answers

Libcurl NXDOMAIN causes timeout

Libcurl in my app seems to have trouble with non existing domains, as it stays a minimum 7-20s on "NXDOMAIN" requests (which seems to match CURLOPT_CONNECTTIMEOUT). here is the pmp (poore man's profiler) output: 2585…
Stefan Rogin
  • 1,499
  • 3
  • 25
  • 41
0
votes
1 answer

Random 500 errors in a multi CURL request

I am trying to get some hal+json data from a web service through a curl_multi to fill in a Bootstrap Typeahead. Every time I run this code, some of my requests in the curl_multi will be 500, and some will return the data I need. The ones that are…
user2185640
  • 23
  • 1
  • 5
0
votes
1 answer

Trouble with PHP cURL proxy and post

I'm making a script that can check if accounts are still valid and working. I'm using this function : function crack() { $url = 'http://127.0.0.1/trying.php'; $agent= 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101…
Kunal Karmakar
  • 573
  • 7
  • 12
0
votes
1 answer

Get cUrl to preemptively send Authorization header for DIGEST authentication in PHP

We have a PHP page (actually a WordPress plugin) which makes calls to a REST API with DIGEST authentication. The curl handle is created on a per request basis, given authorisation credentials and successfully authenticates. PROBLEM: Under the hood,…
Simon Gibbs
  • 4,737
  • 6
  • 50
  • 80
0
votes
0 answers

curl_multi_exec - Stop all calls when one completes

Is it possible to cancel all running calls in curl_multi when one completes successfully? It seems as though curl_multi waits until all processes have finished before finishing the script, but this could be my own perception. I'm attempting to call…
Allan Bogh
  • 605
  • 8
  • 15
0
votes
1 answer

Optimize PHP CURL for web crawler

I am trying to code a crawler based on PHP with curl. I have database of 20,000-30,000 URLs that I have to crawl. Each call to curl to fetch a webpage takes around 4-5 seconds. How can I optimize this and reduce the time required to fetch a page?
Gowtham
  • 386
  • 5
  • 20
0
votes
3 answers

PHP cURL setting a delay after 10 requests

I am using PHP and cURL to scrape the html of a single websites pages. Through experimentation I have discovered that my code only works when I specify 10 URLS or less in the $nodes array(see code sample). I need to scrape around 100 pages at once…
Raj Gundu
  • 147
  • 3
  • 17
0
votes
1 answer

What is the most efficient way to execute a number of php scripts simultaneously?

Currently I'm using multi_curl to execute php files simultaneously on my own server. It seems that it is not a very efficient way as the server gets overloaded when 200+ scripts need to be executed at the same time. I need to send variables to each…
BastiaanWW
  • 1,259
  • 4
  • 18
  • 34
0
votes
1 answer

Why is there a delay in script execution when calling many php scripts simultaneoulsy using multi_curl?

I have a script that calls around 40 individual php scripts (located on the same server) simultaneously. I notice that there is a delay in the start of each individual php script of up to 5 seconds. The server load CPU and memory seem fine, no…
BastiaanWW
  • 1,259
  • 4
  • 18
  • 34
0
votes
1 answer

Making CURL multi requests to the same server as slow as single request

I am using a good Multi CURL interface called Rolling CURL http://code.google.com/p/rolling-curl/issues/detail?id=20 It works fine, for example it gets data from 20 sites in around 3 seconds. The problem is that I need it to work on 200 - 300 sites…
Matthew Underwood
  • 1,459
  • 3
  • 19
  • 33
0
votes
2 answers

How to NOT return data from curl-multi and reduce CPU usage?

I have a script in php that uses multi curl. A general multicurl request consists of 50 individual curl handles. It takes some time for each request to return, so my script should just wait until processing is finished on the remote server. Each…
BastiaanWW
  • 1,259
  • 4
  • 18
  • 34
-1
votes
1 answer

PHP curl_multi_getcontent Always Outputs the content

I am building a site "monitor", and have decided to utilize curl_multi_* due to the number of sites I need to monitor as well as the multiple curl pulls I need to make per iteration. CODE: https://gitlab.com/snippets/1739288 (too much to post here,…
Kevin
  • 2,684
  • 6
  • 35
  • 64
1 2 3
…
13
14