4

I'd like to write a performance-sensitive application in Python, so executing it under PyPy is a natural choice. However, a significant portion of my code depends on numpy, scipy, and scikit-learn. Would it be possible to embed a CPython instance within a running PyPy program in order to call array-oriented code? If not, what's the easiest way to make PyPy and CPython talk to each other?

Tavis Rudd
  • 1,156
  • 6
  • 12
Alex Rubinsteyn
  • 420
  • 4
  • 8
  • 2
    By the way: You may be interested in the "Questions on the pypy+numpy project" thread currently active on the pypy-dev list, which touches on the problems with using SciPy on (current versions of) PyPy and I think someone suggested a workaround to get it working (also using CPython, but in seperate processes). –  Oct 19 '11 at 16:57

2 Answers2

4

Your best bet for the time-being is Cython rather than PyPy. It has c-level performance, if you add type declarations, and excellent integration with numpy, et al.

People are currently working on getting it to work well with PyPy, but that's a ways off yet.

Tavis Rudd
  • 1,156
  • 6
  • 12
  • 3
    Note though: It only has C level performance *if you write C*, just running it over existing Python code does little good. That means restrictions on your code, on the types and APIs you use, carefully placed type annotations and potentially some manual memory management. Of course, Cython is still a very good option, especially regarding NumPy integration. It just isn't the saviour of all Pythonity. –  Oct 19 '11 at 17:00
  • 2
    Yes, I should have mentioned that a lazy code dump from Python to Cython gives you nothing, unlike PyPy. However, you don't need to 'write C'. You write Cython with type declarations and, just maybe, some more attention to memory management. I've found it remarkably easy to get massive speedups just with type declarations in code that is otherwise plain old Python. It still feels like Python. – Tavis Rudd Oct 19 '11 at 17:15
2

No, you can't embed CPython inside PyPy AFAIK. You can, however, use distributed/parallel execution systems to make PyPy talk to CPython. Both execnet and Pyro mention this precise PyPy <-> CPython use case. Other packages from Python Wiki's Parallel Processing page are probably suitable too.

Also, as delnan mentions, there's a current discussion about PyPy developers' plan on implementing Numpy in PyPy (which doesn't include support for scipy, and scikit.learn so far).

TryPyPy
  • 6,214
  • 5
  • 35
  • 63