is refer to multiple synchronous curl transfers
Questions tagged [curl-multi]
197 questions
1
vote
1 answer
php - curl multi, very slow when try get content
trying with curl_multi_init get info for page.
But when i try get info with curl_multi_getcontent() - page after 30s. down.
How i should correctly use curl_multi_getcontent()? Thanks
class Grab
{
public function getData()
{
$sessions…
lolalola
- 3,773
- 20
- 60
- 96
1
vote
1 answer
PHP Multi Curl is Running Very Slow On High Traffic Server Cluster
running into speed issues with multi curl. I am using multi curl to grab XML from various urls, all with response times of under 300ms. But my multi curl function is taking over 1 second to grab these URLS (About 10-15 URLS only). Below is the code…
George Milonas
- 553
- 7
- 22
1
vote
0 answers
Why might a php script run slower in sequence than in separate parts?
I've run into a weird problem, and hope that someone can help me out. I've written a multiCurl spider in PHP that scrapes keywords off websites, and I'm running into a strange performance problem.
When I run the spider to scrape the first few…
Zero Wing
- 233
- 2
- 12
1
vote
0 answers
curl multi stuck in an infinite loop
I am trying to do curl_muliti_*, but I am having issues, and I don't know what is causing the problem. When I run the following code, I get stuck in an infinite loop here while($active && $mrc === CURLM_OK){
$active = null;
do{
$mrc =…
Get Off My Lawn
- 34,175
- 38
- 176
- 338
1
vote
1 answer
PHP cURL multi handling causing random connection issues between servers?
I have a website that tracks individual player's data for an online game. Everyday at the same time a cron is run that uses cURL to fetch each player's data from the game company's server (each player requires their own page to fetch). Previously I…
Capt Otis
- 1,250
- 1
- 12
- 18
1
vote
2 answers
why curl_multi does not work correctly in my localhost?
I use a php function to return the status of more than 1000 websites (i.e. website is up or down).
public function curlCheck($nodes) {
$results = array();
$node_count = count($nodes);
$curl_arr = array();
$master =…
Foad Tahmasebi
- 1,333
- 4
- 16
- 33
1
vote
1 answer
Multiple and Growing Outbound CURL Request
Multi Outbound Request - Curl
Here's the problem, I have several clients I have to CURL outbound requests to. Say for instance I currently have 20 clients and I send around 100~1000 requests per minute to each of them. I also use CURL multi however…
Christian Noel
- 305
- 4
- 14
1
vote
0 answers
PHP & curl_multi and CURLOPT_FILE = No File Contents
when processing calls with the CURLOPT_FILE (sending contents to a file) in parallel (using curl_mult_exec) the file contents when the contents is small (not sure exact size but in this case its 100 - 300 bytes) is always 0.
After the call…
jadent
- 3,674
- 5
- 27
- 32
1
vote
1 answer
How to set a timeout on PHP5 curl calls? Published CURL options do not seem to work
We've written a script that pulls data from an external server. If the server goes down we don't want our server waiting for the data since we process a lot of data and we don't want it bogged down. To address this, we're trying to timeout our…
user77413
- 30,205
- 16
- 46
- 52
1
vote
0 answers
how to run multi threaded curl script from terminal?
I have used a multi curl library for PHP that facilitates fetching multiple pages in parallel (basically an easy to use API).
My Scenario: Fetch user data from API , process it and store results. All those users whose data have to be fetched are…
anasanjaria
- 1,308
- 1
- 15
- 19
1
vote
2 answers
CURLINFO_EFFECTIVE_URL for curl_multi
I'm using PHP's CURL MULTI to download an array of resources in parallel, and I want to get the last effective URL from each of the resources (not just the last one).
Basically, I'm looking for:
curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
for CURL…
Lorien Brune
- 830
- 10
- 16
1
vote
1 answer
Scale multi request to different services
I have a service, where I need to ask 40 external services (API's) to get information from them, by each user request. For example one user is searching for some information and my service is asking 40 external partners to get the information,…
Mutatos
- 1,675
- 4
- 25
- 55
1
vote
0 answers
Broken link check while initiating cron-multi
This code checks broken link of a page by initiating multiple cron. However it is not identifying the correct links which are broken . It always returns 0 broken links though there are broken links available on the page.
$url_list is array of links…
nir
- 83
- 1
- 5
- 11
1
vote
1 answer
PHP - stop all curl_multi requests at a certain point
I have a php script that uses curl_multi to make multiple requests at the same time.
It accesses several of my sites in search for a defined value.
Now the problem that I have is that the curl requests should stop if it found the value I'm searching…
Matthias Dunnowa
- 119
- 1
- 1
- 12
1
vote
1 answer
Why is my CPU high when using many CURL requests which are waiting for response?
I have a large php script that uses both curl and multi- curl requests to my own server and other servers. Besides that it does a number of database requests and other operations on my own server.
The script is too large to put here and won't make…
BastiaanWW
- 1,259
- 4
- 18
- 34