0

I need to run one php file 100000 times at a time. for that i used a exec command in a php file (runmyfile.php) and called that file using putty. The runmyfile.php file is have the following code.

for($i=0;$i<100000; $i++){
exec('php -f /home/myserver/test/myfile.php > /dev/null &');
}

It execute myfile.php file 100000 times in parallel.

This myfile.php fetches rows from mysql database table and perform some calculations and insert this values to another table.

But when running 100000 times it hangs out the server. I'm using centos as server.

Some times I'm getting resource unavailable error too.

If I run it 1000 times it works ok.

when I checked the following ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514889
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1000000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

and my mysql max_connection is 200000

Is there any settings that I need to change. So that I can execute my php file 100000 times properly.

user3378018
  • 21
  • 3
  • 1
    Running 100000 parallel processes is very taxing for the os. Why do you need that? – Mikpa Mar 04 '14 at 07:59
  • Ok. So how many process can I run parallel in centos. I need it for running a cron for doing some operation. The mysql table will have latest 2 billion records. – user3378018 Mar 04 '14 at 09:09
  • That depends on a lot of things, but please explain why. Maybe there is a better solution – Mikpa Mar 04 '14 at 09:11
  • So processing the 2 billion records in one day i need to process latest 1400000 record must be processed in a minuts – user3378018 Mar 04 '14 at 09:13

1 Answers1

1

Maybe you need to redesign your application. If you have the need to process 2 billion records in a Mysql database at a daily basis, I would say that running 100000 scripts in parallel is not the best way.

This would mean that each script process 20000 records, if I understand you correctly. It is not possible to process more records in every script?

Have a look at Big Data

Mikpa
  • 1,912
  • 15
  • 20