14

I was just searching about this new implementation, and i use python 2.7, i must install this, so if i use it, i'll forget the word GIL on CPython?

dano
  • 91,354
  • 19
  • 222
  • 219
Abdelouahab Pp
  • 4,252
  • 11
  • 42
  • 65
  • 1
    http://www.dalkescientific.com/writings/diary/archive/2012/01/19/concurrent.futures.html – Robert Harvey Feb 20 '13 at 23:18
  • 1
    http://docs.python.org/dev/library/concurrent.futures.html#module-concurrent.futures -- Doesn't mention anything about the GIL, so the threads it uses are apparently true threads. Otherwise, why bother with all this hulabaloo? – Robert Harvey Feb 20 '13 at 23:20
  • i've found that asynchronous code loves using concurrent.futures, so i said that it was a medecine for GIL – Abdelouahab Pp Feb 20 '13 at 23:22
  • 1
    @RobertHarvey: You can always build a GreenThreadExecutor that fits the `future` API. (However, this turns out to be a bit tricky, which is why `tulip` will bring a slightly different `future` to the stdlib for use with its coroutine event loop.) – abarnert Feb 20 '13 at 23:22

1 Answers1

35

No, concurrent.futures has almost nothing whatsoever to do with the GIL.

Using processes instead of threads is medicine for the GIL. (Of course, like all medicine, it has side effects. But it works.)

The futures module just gives you a simpler way to schedule and wait on tasks than using threading or multiprocessing directly. And it has the added advantage that you can swap between a thread pool and a process pool (and maybe even a greenlet loop, or something crazy you invent and build) without changing the future code. So, if you don't know whether your code will have GIL problems, you can build it to use threads, and then switch it to use processes with a one-line change, which is pretty nice.

But, if you use the ThreadPoolExecutor, it will have the exact same GIL issues as if you created a thread pool, task queue, etc. manually with threading and queue. If you use the ProcessPoolExecutor, it will avoid the GIL issues in the same way (and with the same tradeoffs) as if you used multiprocessing manually.

And the PyPI package is just a simple backport of the concurrent.futures module from 3.2 to 2.x (and 3.0-3.1). (It doesn't magically give you the new-and-sort-of-improved 3.2 GIL, or the more-improved 3.3 GIL, much less remove the GIL.)

I probably shouldn't even have mentioned the GIL changes, because this seems to have just added confusion… but now, let me try to straighten it out, by oversimplifying terribly.

If you have nothing but IO-bound work, threads are a great way to get concurrency, up to a reasonable limit. And 3.3 does make them work even better—but for most cases, 2.7 is already good enough, and, for most cases where it isn't, 3.3 still isn't good enough. If you want to handle 10000 simultaneous clients, you're going to want to use an event loop (e.g., twisted, tornado, gevent, tulip, etc.) instead of threads.

If you have any CPU-bound work, threads don't help parallelize that work at all. In fact, they make things worse. 3.3 makes that penalty not quite as bad, but it's still a penalty, and you still shouldn't ever do this. If you want to parallelize CPU work, you have to use processes, not threads. The only advantage of 3.3 is that futures is a little easier to use than multiprocessing, and comes built-in instead of needing to install it.

I don't want to discourage you from moving to 3.3, because it's a better implementation of a better language than 2.7. But better concurrency is not a reason to move.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • so they're are different things! it was that GIL was re-written on 3.2 that i seems that it is concurrent.futures that solves the problem, thank you :) – Abdelouahab Pp Feb 20 '13 at 23:23
  • 2
    @AbdelouahabPp: No, `concurrent.futures` does _not_ solve any problem, except the problem of making concurrent code slightly simpler to write. The changes to the GIL in 3.2 and 3.3 are completely independent, and it's just a coincidence that the first significant GIL change in over a decade arrived in the same version of Python as the `futures` library. – abarnert Feb 20 '13 at 23:24
  • so i should then switch to 3.3 if i want to get excellent concurrent code! thank you :) and am sorry because this thread i'll back for it serveral times to get more the idea, am a beginner, and found this library with Tornado – Abdelouahab Pp Feb 20 '13 at 23:30
  • 2
    @AbdelouahabPp: Still no. I mean, yes, you should switch to 3.3. But not to get excellent concurrent code. Let me update the answer. – abarnert Feb 20 '13 at 23:43
  • the avantage of staying on 2.7 is lot of libraries that have not switch to 3.x (like the excellent passlib), i'll try to understand more multiprocessing and try to find the way on 2.7, thank you again :) – Abdelouahab Pp Feb 21 '13 at 00:28
  • 1
    @AbdelouahabPp: First, passlib _has_ switched to 3.x. From https://pypi.python.org/pypi/passlib: "Passlib is a password hashing library for Python 2 & 3." And I just did a `pip-3.3 install passlib` and it works fine. Second, the way to do multiprocessing on 2.7 is almost exactly the same as on 3.3. The `multiprocessing` module is nearly identical between 2.7 and 3.3, and the PyPI `futures` backport is nearly identical to the 3.3 `concurrent.futures` in the stdlib. – abarnert Feb 21 '13 at 00:37
  • so i'll stick then with 2.7, there will be an update in some days ;) – Abdelouahab Pp Feb 21 '13 at 00:46
  • 1
    @AbdelouahabPp: I don't understand what you mean. If you're sticking with 2.7 until `passlib` is available for 3.3, you don't need to do wait. It's already available for 3.3. – abarnert Feb 21 '13 at 00:54
  • i just want to leave to 3.x because i think one day, the migration will be, because what i see as new for 3.x compared to 2.x, pushs me to switch to 3.x, so i'll prepare myself to swithc one day :D – Abdelouahab Pp Feb 21 '13 at 01:02
  • and i just remember, Passlib has not BCrypt support for python 3.x https://code.google.com/p/passlib/issues/detail?id=40&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary%20Stars – Abdelouahab Pp Feb 21 '13 at 01:08