4

I have a fairly memory expensive Python program to run on a computer that has 8 CPUs (using Python 3.1 64 bit). The problem is that it uses all 8 processors to 100% usage and thus freezes the machine and makes things slow. How would I make Python use only 7 of the processors to free up more CPU? I have experimented with the Pool class within the multiprocessing library many times, but no luck.

For instance, if I want to call the main function of my program and have it use only 7 processors, it seems like all I would have to do is the following, but no luck:

from multiprocessing import Pool Pool(7, main())

Can anybody tell me where I am going wrong? Thanks!

Chad Kimmel
  • 41
  • 1
  • 2
  • When I said "processors", I really meant "CPUs". Sorry for my bad lingo. – Chad Kimmel Mar 31 '11 at 04:12
  • Do you have any other multiprocessing in your program? If not, python will not use more than one core at a time, this is something you have to purpose to do. Also, you said memory expensive, are you swapping? I'm not sure if this would spike cpu, but it's something to look into. – Adam Wagner Mar 31 '11 at 04:18
  • No, I don't have any other multiprocessing in my program. I am using just one process really. The problem is that this one process is using all 8 cores, and I want it to use only 7. Thanks! – Chad Kimmel Mar 31 '11 at 04:49
  • Adam is right: To use more than one core you need to do some sort of multiprocessing or calls to other programs. It will not use all 8 cores by itself. – Lennart Regebro Mar 31 '11 at 05:21
  • That is what you would think, but it's showing that it is using all 8 cores. Do you know what direction I could go in coding wise? – Chad Kimmel Mar 31 '11 at 06:10

3 Answers3

2

I had the same problem, which I fixed by allocating one less to the pool:

pool= multiprocessing.Pool(processes=(multiprocessing.cpu_count() - 1))

By doing this and letting your pool only use CPU-1, you can save some extra power for your PC. This helped tremendously on my windows machine so I could still click around into other applications (W7 64 bit). On Linux, the machine was OK either way.

Thomas
  • 305
  • 1
  • 2
  • 11
0

I'm not quite sure about your situation, but one thing which might work for you, is to find out number of cores you have ( in python you can use multiprocessing library ), and then limit the number of threads/processes that your program creates to less than one of this number .

Rsh
  • 7,214
  • 5
  • 36
  • 45
0

You are somehow creating a lot of processes. This can be done by the os.system(), os.popen*(), os.spawn*() or methods in the popen2, subprocess and multiprocessing libraries.

If you aren't using CPython but another Python implementation such as Jython or Stackless, you can also use up many cores by threading.

You'll have to figure out what you are doing that is causing this, and then make sure you don't do more than 7 of those at once.

Lennart Regebro
  • 167,292
  • 41
  • 224
  • 251
  • 1
    Actually, I figured out the solution, it was easy all along. Using windows, if you go to the task manager and you right click on the Python Process, and then click on "Set Affinity", you can control the number of processes that Python is using directly through the OS. Lots of headaches solved. Thanks guys! – Chad Kimmel Mar 31 '11 at 16:36
  • @Chad: A right, if you don't mind that the setting is global that works, yeah. – Lennart Regebro Mar 31 '11 at 19:37