I have a PHP script which I am running on an Ubuntu linux machine. The script spawns multiple processes using the pcntl_fork() function, and uses the pcntl_waitpid() function to log that they are killed. It spawns these VERY often (I estimate about 40-50/second), but these processes are each killed immediately (I have tried both exit() and posix_kill(${pid}, SIGKILL), to no avail). The script works fine for several seconds (depending, 10~30 seconds), but inevitably halts and stops creating the 'children' processes. The memory usage on the machine due to the script does not increase, but when the script halts, the cpu on the machine slowly spikes until I force kill the script with Ctrl-C. Each process is meant to parse a line of text and finally save it to a file. For testing purposes, I am simply exiting the child processes as soon as they are created. In one test around 1400 processes were successfully started and killed before the script froze, though like I said this ranges.
I understand that a machine has a ulimit, but I believe I read that it limits the number of concurrent processes. As this script kills the children processes as soon as they are created, I am confused as to what is happening. Here is the output from my current ulimit configuration (ulimit -a):
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 29470 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 29470 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Is there a global limit in PHP that determines the total number of processes created during a script execution? **Keep in mind that these children processes are killed immediately, so I don't believe this is an issue of an unlimited number of processes being created that are stealing system resources.
Here is some source code:
I initialize the fork with this
$this->process_control->fork(array($this, "${FUNCTION_NAME}"), array(STRING_LINE), false);
The process forking function. In this instance $callback is the name of the function to call in the appropriate class. $params is an array of parameters to pass to the function specified in $callback.
public function fork ($callback, $params = null, $hang = true)
{
$this->logger->write_log('log', "Entered fork function!");
// Evaluate the return value of the fork
switch ($pid = pcntl_fork()) {
case -1: // Failed
$this->logger->write_log('error', "Could not fork!");
exit(1);
break;
case 0: // Child created succesfully
$this->logger->write_log('log', "Entered child function!");
$this->logger->write_log('log', 'child ' . posix_getpid() . ' started');
if (empty($callback)) {
$this->logger->write_log('warn', "Callback empty, nothing to do!");
exit(1);
}
if (is_array($callback) && is_array($params)) {
if (!call_user_func_array($callback, $params)) {
$this->logger->write_log('error', "Daemonized process returned false!");
exit(1);
} else {
exit(0);
}
} else {
if (!call_user_func($callback, $params)) {
$this->logger->write_log('error', "Daemonized process returned false!");
exit(1);
} else {
exit(0);
}
}
break;
default: // Parent
$this->logger->write_log('log', "Entered parent function!");
if ($hang != true) {
$this->wait($pid, false);
} else {
$this->wait($pid);
}
break;
}
}
public function wait($p_id, $hang = true)
{
if ($hang) {
$pid = pcntl_waitpid($p_id, $status);
} else {
$pid = pcntl_waitpid($p_id, $status, WNOHANG);
}
switch($pid) {
case -1:
case 0:
$this->logger->write_log('log', "child exited");
break;
default:
$this->logger->write_log('log', "child $pid exited");
break;
}
}
The function that actually does the processing of the lines of text. The lines of text are JSON objects:
public function FUNCTION_NAME($line) {
$this->logger->write_log('info', 'entered FUNCTION_NAME function');
$start_time = microtime(true);
try {
# check to see that the JSON line is not malformed
$line_array = json_decode($line, true);
if (!isset($line_array)) {
throw new Exception('Could not successfully process line');
}
# save the contents to disk
if (!file_put_contents(FILE_NAME, $line, LOCK_EX)) {
throw new Exception('file could not be saved');
}
$this->logger->write_log('info', 'saved line');
return true;
} catch (Exception $e) {
$this->logger->write_log('error', $e->getMessage());
$this->logger->write_log('error', '----------------------------------------------------');
$this->logger->write_log('error', var_export($line, true));
$this->logger->write_log('error', '----------------------------------------------------');
file_put_contents(ERROR_SRC_FILE, $line, FILE_APPEND);
return false;
}
}
Sorry if this is too much code, let me know of any questions
