2

Following up on David Beazley's paper regarding Python and GIL, would it be a good practice to limit a Python program (CPython with GIL and all) to a single CPU in a Windows based multi-core system?

Would it boost performance?

UPDATE: Assume multiple threads are used (not sure if it makes a difference)

Jonathan Livni
  • 101,334
  • 104
  • 266
  • 359
  • 1
    CPython can use only one CPU - so why "limit" to a single CPU??? –  Jul 12 '11 at 16:39
  • Yea, Blackmoon tis right. If you want to use mutliple cores then you have to use something like the multiprocessing module. – pokstad Jul 12 '11 at 16:54
  • @Blackmoon, from reading the linked paper, I don't think that's true. Specifically checkout page 37. – Jonathan Livni Jul 12 '11 at 16:55
  • I don't need to read a paper - this fact with the GIL and CPython is true since the very first of CPython in 1993 and I am using CPython since 1993. Your assumption is wrong or you are misreading something. –  Jul 12 '11 at 17:10
  • The paper is concerning threads, so Jonathan needs to mention that in the question. – pokstad Jul 12 '11 at 17:12
  • Does this make a difference? CPython is limited to one CPU only right now. –  Jul 12 '11 at 17:17
  • 1
    It is limited to a single thread running at any given moment (unless using extensions or other tricks), but I'm not sure the OS necessarily runs them on the same CPU, incurring a penalty. I may have misunderstood David's paper, in which case please correct my understanding of it – Jonathan Livni Jul 12 '11 at 17:22

2 Answers2

4

The paper does indeed imply that limiting a program to a single core would improve performance in that particular case. However, there are a number of concerns that you would need to deal with:

  1. His tests are mainly for compute intensive threads rather than IO bound threads. If the threads you are using often block voluntarily (such as in a web server waiting for a client) then you don't run into GIL issues at all.
  2. The GIL issues deal specifically with threads and not processes. I may be reading your question wrong, but you seem to be asking about restricting all Python programs to a single core. Programs using processes for parallelism don't suffer from GIL issues and restricting them to a single core will make them slower.
  3. The GIL is drastically different in Python 3.2 (as David mentions in this video. The GIL was changed explicitly to deal with such issues. While it still has problems, it no longer has this problem.

In summary, the only time you want to complicate your life by forcing the OS to restrict the program to a single core is when you are running a:

  1. Multithreaded
  2. Compute Intensive
  3. Lower than Python 3.2

program on a multicore machine.

Abhay Buch
  • 4,548
  • 1
  • 21
  • 26
  • +1 I recently took Beazley's concurrency class (highly recommend) and Python32 came up. It contains one of the first major changes to multithreading and the GIL in a looooong time, and arguably a bad one. – pokstad Jul 12 '11 at 20:30
0

Bias : For parallel computing involving heavy CPU processing, I much prefer message passing and cooperating processes to thread programming (of course, it depends on the problem)

You shouldn't limit your programs to one core. Beazley was just demonstrating a specific problem that performed poorly under those unque conditions (those conditions being an IO bound thread and a CPU bound thread competing against each other). Ideally you want to avoid those conditions by using a different method (import multiprocessing).

I think the best solution is to put your CPU bound tasks in other processes using the multiprocessing module so that they utilize their own cores, and IO bound tasks in threads (or microthreads/coroutines, if you read his interesting paper on that: http://www.dabeaz.com/coroutines/) since the GIL is best suited for those types of tasks.

Conclusion: Python threads are best suited for IO bound tasks, NOT CPU bound.

pokstad
  • 3,411
  • 3
  • 30
  • 39