0

I'm currently trying to process a bunch of data with apache and php,
I estimate it will take a few hours and I'd like to reduce that time

In my webmin I see following information

CPU usage   25% user, 0% kernel, 19% IO, 55% idle

is there a way i can dedicate more cpu usage to the process temporarily? (even if it means changing configs and restarting apache twice)

it's basically a lot of mysql selects/inserts/updates and regex

The processor is Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz, 4 cores

Thanks for any tips on how to improve the speed

edit The CPU usage list shows this

ID      Owner       CPU     Command   
26841   mysql       38.6 %  /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file ...
5627    www-data    0.3 %   /usr/sbin/apache2 -k start
Moak
  • 734
  • 3
  • 10
  • 31

2 Answers2

1

Given that you've got a quad core CPU and you're using exactly a quarter of it, I'd be inclined to say that you're running all your data processing in one single-threaded job. This will only use one CPU core. Rewrite your code to execute several tasks in parallel (either through threads or multiple independent processes) and it will use all of your CPU cores correctly. This is not a trivial operation.

Note that questions regarding parallelising your code should be directed to Stack Overflow.

womble
  • 96,255
  • 29
  • 175
  • 230
  • it seems to be mostly dependant on MySQL, but is it possible to have multiple mysql processes? – Moak Jul 08 '11 at 09:26
  • If you mean mysqld on different ports, then no. I am more than sure, it is not possible, as you would do sharing violation within database files. – John Jul 08 '11 at 09:53
  • No, you only run one MySQL server, but that's multithreaded so it'll use multiple CPU cores if they're available. But if you're only issuing one query at a time, it can only do one thing at a time. Again, parallelise your data processing to better utilise your system resources. – womble Jul 08 '11 at 10:16
1

MySQL from version 5.0 is already multi-threaded and optimized for running on multi-core systems. Is your disk subsystem running well ? Maybe the disk is representing the bottleneck.

Otherwise, you have to recode your PHP scripts in order to run as multiple parallel processes as stated above. PHP does not fully support threading within one instance, or it is struggle to use it.

@edit: Or you can divide data to some blocks if it is possible and process it by running multiple different PHP scripts.

John
  • 1,493
  • 3
  • 26
  • 46