Why is my CPU high when using many CURL requests which are waiting for response?

Question

I have a large php script that uses both curl and multi- curl requests to my own server and other servers. Besides that it does a number of database requests and other operations on my own server.

The script is too large to put here and won't make things easier.

I measured the time throughout the whole script. I made a distinction between the time required for operations on my own server and operations on an external server.

It appeared that adding up the elapsed time for all operations on my own server it all together never takes longer than 0.5 seconds. Therefore I expect that my own script is not heavy for my server. The total elapsed time is 25 seconds on average.

The majority of the time: 24.5 seconds my server is waiting for responses from external servers. My server does make around 300 curl requests, but a curl request on itself is not heavy.

The problem: When I run the script the cpu goes to 95% and the server is (nearly) not responding for other requests as long as this script is running.

This is an example of a multi-curl request:

<?PHP
// Multi curl:
$imax=20;
// build the individual requests as above, but do not execute them
for ($i=1; $i<$imax; $i ++ ):
${'ch_'.$i} = curl_init('http://www.domain.com/');
//$ch_2 = curl_init('http://www.domain.com/');
curl_setopt(${'ch_'.$i}, CURLOPT_RETURNTRANSFER, true);
//curl_setopt($ch_2, CURLOPT_RETURNTRANSFER, true);
endfor;

// build the multi-curl handle, adding both $ch
$mh = curl_multi_init();
for ($i=1; $i<$imax; $i ++ ):
curl_multi_add_handle($mh, ${'ch_'.$i});
endfor;

// execute all queries simultaneously, and continue when all are complete
$running = null;
do {
curl_multi_exec($mh, $running);
} while ($running);

// all of our requests are done, we can now access the results
for ($i=1; $i<$imax; $i ++ ):
${'response_'.$i} = curl_multi_getcontent(${'ch_'.$i});
endfor;
//$response_2 = curl_multi_getcontent($ch_2);

for ($i=1; $i<$imax; $i ++ ):
echo "${'response_'.$i}"; // same output as first example
endfor
?>

httpd.conf:

#
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.2> for detailed information.
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  

ServerRoot "/etc/httpd"
Listen 80

#LoadModule dummy_module /usr/lib/apache/mod_dummy.so
#LoadModule php5_module        /usr/lib/apache/libphp5.so
Include /etc/httpd/conf/extra/httpd-phpmodules.conf

User apache
Group apache

KeepAlive On
KeepAliveTimeout 10

ServerAdmin admin@localhost
DocumentRoot "/var/www/html"

MaxClients 10000
MaxRequestsPerChild 50000

<Directory /home/*>
    AllowOverride All
    Options -MultiViews -Indexes FollowSymlinks IncludesNoExec +Includes
<Limit GET POST OPTIONS PROPFIND>
    Order allow,deny
    Allow from all
</Limit>
<LimitExcept GET POST OPTIONS PROPFIND>
    Order deny,allow
    Deny from all
</LimitExcept>
</Directory>

<Directory />
    Options All
    AllowOverride All
</Directory>

<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all
   <IfModule mod_suphp.c>
        suPHP_Engine On
        suPHP_UserGroup webapps webapps
    SetEnv PHP_INI_SCAN_DIR
   </IfModule>
</Directory>

<IfModule mod_dir.c>
    DirectoryIndex index.html index.htm index.shtml index.php index.php5 index.php4 index.php3 index.phtml index.cgi
</IfModule>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

ErrorLog /var/log/httpd/error_log
LogLevel warn

#replace %b with %O for more accurate logging
<IfModule mod_logio.c>
    LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %O" common
    LogFormat "%O %I" bytes
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>

    CustomLog /var/log/httpd/access_log common

<IfModule mod_alias.c>
    # Include some DirectAdmin alias
    Include conf/extra/httpd-alias.conf
</IfModule>

<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

DefaultType text/plain

<IfModule mod_mime.c>
    TypesConfig conf/mime.types
    AddType application/x-gzip .tgz
    AddEncoding x-compress .Z
    AddEncoding x-gzip .gz .tgz
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddHandler cgi-script .cgi
    AddHandler type-map var
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
    AddType video/x-ms-asf .avi
    AddType video/mpeg .mpg
    AddType video/mpeg .mpeg
    AddType video/quicktime .mov
    AddType video/x-ms-wmv .wmv
</IfModule>

#EnableMMAP off
#EnableSendfile off

#######################################################################################
# Do not change anything in included files, because they are rewritten by DirectAdmin #
#######################################################################################

# This is needed for PHP
Include conf/extra/httpd-php-handlers.conf

# Server-pool management (MPM specific)
Include conf/extra/httpd-mpm.conf

# Multi-language error messages
Include conf/extra/httpd-multilang-errordoc.conf

# Fancy directory listings
Include conf/extra/httpd-autoindex.conf

# Language settings
Include conf/extra/httpd-languages.conf

# User home directories
#Include conf/extra/httpd-userdir.conf

# Real-time info on requests and configuration
Include conf/extra/httpd-info.conf

# Virtual hosts
Include conf/extra/httpd-vhosts.conf

# Local access to the Apache HTTP Server Manual
#Include conf/extra/httpd-manual.conf

# Distributed authoring and versioning (WebDAV)
Include conf/extra/httpd-dav.conf

# Various default settings
Include conf/extra/httpd-default.conf

# Secure (SSL/TLS) connections
Include conf/extra/httpd-ssl.conf

# Deflate module settings
Include conf/extra/httpd-deflate.conf

# All the DirectAdmin vhosts
Include conf/extra/directadmin-vhosts.conf

# All suPHP directives
Include conf/extra/httpd-suphp.conf

# For user configurations not maintained by DirectAdmin. Empty by default.
Include conf/extra/httpd-includes.conf

#######################################################################################
# End of included files that are rewritten by DirectAdmin                             #
#######################################################################################

<IfModule mod_ssl.c>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>

My question: How can I better configure my server to prevent overload?

Your Apache config has nothing to do with this. Post some code! I suspect you have an infinitely loop to try to emulate a block without a sleep. No need to post your whole program... just the relevant parts. Break it down to a reproducible example. — Brad, May 09 '12 at 21:08
@Brad I tested the program for a case where it didn't have to wait for response from the external server, and in that case it worked very quick without the CPU going crazy. — BastiaanWW, May 09 '12 at 21:15
Yes, that supports what I just said. Can you post an example? — Brad, May 09 '12 at 21:16
@Brad I posted an example of a Multi curl I within the program — BastiaanWW, May 09 '12 at 21:25

score 0 · Answer 1 · answered May 09 '12 at 21:30

0

Just a guess: curl-multi-exec returns a value that you can (and should) check in the while block

do {
    $mrc = curl_multi_exec($mh, $active);
} while ($active && $mrc == CURLM_CALL_MULTI_PERFORM);

As stated in the PHP documentation

answered May 09 '12 at 21:30

Alfabravo

7,493
6
46
82

Why is my CPU high when using many CURL requests which are waiting for response?

1 Answers1