High CPU usage with php command line scripts, is php-fpm the solution?

Question

Update (2013 Oct, 1st):

Today I finally found out what was the issue. After debugging for hours with xdebug and trying to execute multiple scripts I noticed that the body of the script is not the problem. When executing tests worker with a sleep of 10-20 seconds I've noted that the CPU was idling most of the time and so deducted that was consuming the most of the CPU is to bootstrap Symfony.

My scripts were very quickly executed and killed to pawn to new script, etc. I've fixed it adding a do{}while() that is exiting after a random amount of seconds (to avoid all the worker to restart at the same time).

I've reduced the load from an average of 35-45% to an average of 0.5-1.5% That's a HUGE improvement. To Resume Symfony is bootstrapped once and after the script is just waiting until a random timeout to kill itself and launch a new instance of itself. This is to avoid script to hang or the database connection to timeout, etc.

If you have a better solution do not hesitate to share. I'm so happy to go from 100% CPU usage (x4 servers because of the auto-scaling) to less than 1% (and only one server) for the same amount of work, it's even faster now.

---

Update (2013 Sep, 24th):

Just noticed that the console component of Symfony is using dev environment by default. I've specified prod in the command line: ./app/console --env=prod my:command:action and I divide by 5 the execution time which is pretty good.

Also I have the feeling that curl_exec is eating a lot of CPU but I'm not sure.

I'm trying to debug the CPU usage using xdebug, reading the generated cachegrind, but there is no reference of CPU cycle used per function, class, ... Only the time spent and memory used.

If you want to use xdebug in a PHP command line just use #!/usr/bin/env php -d xdebug.profiler_enable=On at the top of the script

If anyone has a tip to debug this with xdebug I'll be happy to hear it ;)

---

I'm asking this question without real hope.

I have a server that I use to run workers to process some background tasks. This server is an EC2 server (m1.small) inside an auto-scaling group with high CPU alert setup.

I have something like 20 workers (php script instance) waiting for jobs to be processed. To run the script I'm using the console component of Symfony 2.3 framework.

There is not much happening in the job, fetching data from URL, looping over the results and insert it row by row (~1000 rows per job) in MySQL (RDS server).

The thing is that with 1 or 2 workers running, the CPU is at 100% (I don't think it's like at 100% all the time but it's spiking every second or so) which cause the auto-scaling group to launch new instances.

I'd like to reduce the CPU usage which is not justified at all. I was looking at php-fpm (fastCGI) but it looks like it's for web servers only. PHP client wouldn't use it? right?

Any help would be appreciated, Cheers

Answer 1

I'm running PHP 5.5.3 with FPM SAPI and as @datasage pointed out in his comment this would only affect the web-based side of things. You run a php -v command on CLI you'd notice:

PHP 5.5.3 (cli) (built: Sep 17 2013 19:13:27)

So FPM isn't really part of the CLI stuff.

I'm also running a similar situation which you are, except I'm running jobs via Zend Framework 2. I've found that running jobs which loop over information can be resource intensive at times but I've also found that it was caused by the way I had originally developed that loop myself. It had nothing to do with PHP in general itself.

I'm not sure about your setup, but in one of my jobs which runs forever I've found this works out the best and my server load is almost null.

[root@router ~]$ w
12:20:45 up  4:41,  1 user,  load average: 0.00, 0.01, 0.05
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0     11:52    5.00s  0.02s  0.00s w

Here is just an "example":

do {
    sleep(2);

    // Do your processing here
} while (1);

Where // Do your processing here I'm actually running several DB Queries, processing files, and running server commands based on the job requirements.

So in short, I wouldn't blame or think that PHP-FPM is causing your problems but most likely I'd start looking at how you've developed your code to run and make the necessary changes.

I currently have four jobs right now which run forever and is continuously looking at my DB for jobs to process and the server load has never spiked. I've even tested this with 1,000 jobs pending.