26

Google is sponsoring an Open Source project to increase the speed of Python by 5x.

Unladen-Swallow seems to have a good project plan

Why is concurrency such a hard problem?
Is LLVM going to solve the concurrency problem?
Are there solutions other than Multi-core for Hardware advancement?

Don Wakefield
  • 8,693
  • 3
  • 36
  • 54
lprsd
  • 84,407
  • 47
  • 135
  • 168
  • 2
    That's not one question. I see at least three, possibly four. – Devin Jeanpierre Mar 29 '09 at 21:04
  • Would you have me ask all separate questions? They are all related, in a way. Aren't they? :) – lprsd Mar 29 '09 at 21:09
  • "Py3k seems to have solved Memory Footprint problem (by returning iterators than lists); Now if LLVM solves speed issue?" what is the question? There is some sort of implied question... but it is not clear what it is. And yes, there are too many questions in one here! – TofuBeer Mar 29 '09 at 21:10
  • @TofuBeer: Edited the last part, and removed it, It is indeed perhaps suited for a different question – lprsd Mar 29 '09 at 21:23

3 Answers3

32

LLVM is several things together - kind of a virtual machine/optimizing compiler, combined with different frontends that take the input in a particular language and output the result in an intermediate language. This intermediate output can be run with the virtual machine, or can be used to generate a standalone executable.

The problem with concurrency is that, although it was used for a long time in scientific computing, it has just recently has become common in consumer apps. So while it's widely known how to program a scientific calculation program to achieve great performance, it is completely different thing to write a mail user agent/word processor that can be good at concurrency. Also, most of the current OS's were being designed with a single processor in mind, and they may not be fully prepared for multicore processors.

The benefit of LLVM with respect to concurrency is that you have an intermediate output, and if in the future there are advances in concurrency, then by updating your interpreter you instantly gain those benefits in all LLVM-compiled programs. This is not so easy if you had compiled to a standalone executable. So LLVM doesn't solve the concurrency problem per se but it leaves an open door for future enhancements.

Sure there are more possible advances for the hardware like quantum computers, genetics computers, etc. But we have to wait for them to become a reality.

BlueRaja - Danny Pflughoeft
  • 84,206
  • 33
  • 197
  • 283
Ismael
  • 2,995
  • 29
  • 45
  • 3
    @Ismael: I don't think it's a matter of scientific vs. non-scientific applications. The real issue is language and OS support for concurrent programming is bad, and some folks are willing to work harder to overcome the bad support for concurrency. – S.Lott Mar 30 '09 at 13:10
  • @Lott: Yes, you are right. Although there are tools to program with concurrency in mind, they are hard to use and will require a significant effort. – Ismael Mar 30 '09 at 15:13
17

The switch to LLVM itself isn't solving the concurrency problem. That's being solved separately, by getting rid of the Global Interpreter Lock.

I'm not sure how I feel about that; I use threads mainly to deal with blocking I/O, not to take advantage of multicore processors (for that, I would use the multiprocessing module to spawn separate processes).

So I kind of like the GIL; it makes my life a lot easier not having to think about tricky synchronization issues.

DNS
  • 37,249
  • 18
  • 95
  • 132
  • 9
    If they can get rid of the GIL but make sure that everything you expect to be atomic is still atomic, then I think it should at worst have no effect on you, and could improve your apps' performance in some cases. – John Fouhy Mar 30 '09 at 00:18
15

LLVM takes care of the nitty-gritty of code generation, so it lets them rewrite Psyco in a way that's more general, portable, maintainable. That in turn allows them to rewrite the CPython core, which lets them experiment with alternate GCs and other things needed to improve python's support for concurrency.

In other words, LLVM doesn't solve the concurrency problem, it just frees up your hands so YOU can solve it.

Rhamphoryncus
  • 339
  • 1
  • 6