4

I have a Python project where performance is fairly important. Honestly, I know Python (at least pure Python) isn't the best tool for the job, but I'm using it anyhow because I want to distribute a very simple pure python version of some research code so that people can modify it, etc. without having to set up a complicated environment. (I also have an alternative implementation, written in D, which is much faster but harder to hack.)

I was using PyPy as my interpreter and recommending it to anyone who usd this code. This was giving me reasonable speed. I then decided to tweak the algorithm and needed to access to some functions from SciPy (all I need is the CDF, survival function and inverse CDF for the Chi-square distribution). I tried running my code in CPython to get access to SciPy and it's slow as molasses.

Can someone suggest one of three things:

  1. A kludge that will allow me to use some bits and pieces of SciPy from PyPy.

  2. Even better, pure python implementations of the chi-square distribution functions I need, under a liberal license?

  3. A way to JIT the bottlenecks from CPython? I'm aware of Psyco, but it appears unmaintained and only works on 32-bit systems.

dsimcha
  • 67,514
  • 53
  • 213
  • 334
  • I've found that implementations of statistics code in python are more often than not direct transliteration of the code in Numerical Recipes (http://www.nr.com/aboutNR3book.html). So, if you can't find the Chi-Squared distribution function, there's always the transliteration approach. Chapter 6 here: http://www.nrbook.com/a/bookcpdf.php. – Wilduck Dec 07 '11 at 19:30
  • @Wilduck: I'm aware of the translating option. As a last resort I might translate what I need from the D standard library, which has what I need. It's just that this is complicated numerics code that might be almost as long as my main program, so I'd like to avoid doing so. – dsimcha Dec 07 '11 at 19:43

1 Answers1

5

SciPy does not work on PyPy yet, although there is some work to make it happen relatively soonish, provided you're willing to use nightly and make little adjustments so you use only the supported subset of numpy (that expands rapidly anyway). I would suggest you simply try porting the SciPy functions to Python for now if time rushes you (they should be relatively simple, but still work). There is also a numpy fundraising on PyPy website to make this happen even sooner :)

EDIT: This is matplotlib running on pypy by embedding CPython: http://baroquesoftware.com/~fijal/extreme_hacks.png

EDIT2: read http://morepypy.blogspot.com/2011/12/plotting-using-matplotlib-from-pypy.html for an example.

Cheers, fijal

fijal
  • 3,190
  • 18
  • 21