What is f2py used for while building numpy source?

Question

When I list all the Fortran files in NumPy's source tree, I get:

./doc/source/f2py/scalar.f
./doc/source/f2py/string.f
./doc/source/f2py/calculate.f
./doc/source/f2py/moddata.f90
./doc/source/f2py/array.f
./doc/source/f2py/allocarr.f90
./doc/source/f2py/extcallback.f
./doc/source/f2py/common.f
./doc/source/f2py/ftype.f
./doc/source/f2py/fib3.f
./doc/source/f2py/callback.f
./doc/source/f2py/fib1.f
./doc/f2py/f2python9-final/src/examples/exp1.f
./doc/f2py/simple.f
./doc/f2py/multiarray/foo.f
./doc/f2py/hello.f
./doc/f2py/ex1/bar.f
./doc/f2py/ex1/foobar-smart.f90
./doc/f2py/ex1/foo.f
./doc/f2py/ex1/arr.f
./doc/f2py/ex1/foobar.f90
./numpy/f2py/tests/src/mixed/foo_fixed.f90
./numpy/f2py/tests/src/mixed/foo_free.f90
./numpy/f2py/tests/src/mixed/foo.f
./numpy/f2py/tests/src/size/foo.f90
./numpy/f2py/tests/src/kind/foo.f90
./numpy/f2py/tests/src/assumed_shape/precision.f90
./numpy/f2py/tests/src/assumed_shape/foo_use.f90
./numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap
./numpy/f2py/tests/src/assumed_shape/foo_free.f90
./numpy/f2py/tests/src/assumed_shape/foo_mod.f90
./numpy/f2py/src/test/bar.f
./numpy/f2py/src/test/foo.f
./numpy/f2py/src/test/foo90.f90
./numpy/f2py/src/test/wrap.f
./numpy/distutils/tests/f2py_ext/src/fib1.f
./numpy/distutils/tests/f2py_f90_ext/include/body.f90
./numpy/distutils/tests/f2py_f90_ext/src/foo_free.f90

So none other than f2py itself uses Fortran. I looked into the linear algebra modules. For LAPACK, there is a make_lite.py file which extracts only the necessary subroutines from a LAPACK source tree, and converts them into C using f2c. So when exactly during the creation of NumPy was it convenient to create f2py? Am I missing something?

EDIT

It turns out a lot of packages inside SciPy use f2py. Running

$ find . -iname '*.f*'  | cut -d'/' -f3,4 | uniq

gives me the exact directories filled with Fortran files:

linalg/src
fftpack/src
odr/odrpack
special/cdflib
special/amos
special/mach
special/specfun
integrate/quadpack
integrate/odepack
integrate/dop
integrate/linpack_lite
integrate/mach
sparse/linalg
interpolate/fitpack
optimize/minpack2
optimize/minpack
optimize/nnls
optimize/cobyla
optimize/lbfgsb
optimize/slsqp
stats/mvndst.f
stats/futil.f
stats/statlib
_build_utils/src
lib/lapack

Notice where those Fortran files are - they are all in either `/doc/` or `/test/` directories. None of them are components of `f2py` itself, which seems to be written in Python and C. — ali_m, May 14 '14 at 23:24
I have a feeling that `f2py` is actually a requirement for building scipy rather than numpy. Take a look at all of the Fortran files in the scipy source tree - tons of critical stuff including `fftpack`, `odepack`, `arpack`, `fitpack`, etc... — ali_m, May 14 '14 at 23:55

ali_m · Accepted Answer · 2014-05-15T02:03:07.767

OK, after a bit of poking around I think I've confirmed a few of my initial suspicions

First:

So none other than f2py itself uses Fortran.

As I mentioned in the comments, all of the Fortran source files the OP is referring to are in /test/ or /doc/ directories, and I therefore suspect that they are for testing and documenting f2py (and numpy.distutils, which uses f2py). A glance at a few of the source files seems to confirm this impression. f2py itself looks like it's written in Python and C.

I looked into the linear algebra modules. For LAPACK, there is a make_lite.py file which extracts only the necessary subroutines from a LAPACK source tree, and converts them into C using f2c

This seemed strange to me, since I don't actually have f2c installed (or Plex, which is another library that seems to be required by make_lite.py). I decided to stick an extra line in main() to show whether make_lite.py actually gets used during a normal install:

...
def main():

    # let's see if you're actually doing anything
    import subprocess; subprocess.call(['touch', '/tmp/hello_from_make_lite'])
...

Sure enough, after installing numpy in a clean virtualenv there is no hello_from_make_lite file in my /tmp/, indicating that make_lite.main() never executed. Take a look at numpy/linalg/lapack_lite/README:

The numpy/linalg/blas_lite.c, numpy/linalg/dlapack_lite.c, and numpy/linalg/zlapack_lite.c are f2c'd versions of the LAPACK routines required by the LinearAlgebra module, and wrapped by the lapack_lite module. The scripts in this directory can be used to create these files automatically from a directory of LAPACK source files.

So numpy is already distributed with these f2c'd C source files - there's no need to use make_lite.py unless you're a dev wanting to update these functions from a new version of the LAPACK library.

So when exactly during the creation of NumPy was it convenient to create f2py?

As far as I can tell, f2py doesn't get used at all during a normal numpy install. Again, I stuck an extra line in f2py2e.main():

...
def main():
    import subprocess; subprocess.call(['touch', '/tmp/hello_from_f2py2e'])
...

And again, /tmp/hello_from_f2py2e doesn't exist after a normal install of numpy.

So what is f2py actually used for? Check out the scipy source tree, and from its root call

$ find . -iname *.f*

You'll see loads and loads of important-looking Fortran files, including fftpack, odepack, arpack, fitpack, etc. I suspect that f2py is mainly required to wrap the Fortran extensions for scipy rather than numpy.

I might be wrong, though - perhaps one of the numpy or scipy devs will set me straight.

Update

Actually, I think f2py is not actually required during a normal installation of scipy either! If you take a look in the source directory for one of the Fortran modules, e.g. fftpack, you'll see that it already contains .pyf files, which would normally be automatically generated by f2py and define the interfaces for the Fortran functions (see here).

I think the deal is that f2py was used to initially generate the .pyf wrappers for the Fortran functions, but these .pyf files are distributed along with the rest of the source tree so that it's unnecessary to run f2py again during the normal build process.

A modified version of f2c is present under linalg/lapack_lite — osolmaz, May 15 '14 at 06:43
Looking at scipy source, I see many packages with Fortran files. — osolmaz, May 15 '14 at 21:57
I'm beginning to think f2py is dead code. The documentation pages on the web all cyclically reference one another, if not referencing a dead link, and it appears, with what few coherent f2py pages that exists, that the code hasn't been touched in 5 years, if not longer. — EMiller, Dec 04 '16 at 18:54

score 0 · Answer 2 · answered Feb 16 '21 at 22:59

This is an old question, but the purpose of f2py is to allow users of numpy to wrap their own arbitrary Fortran routines and call them from within Python. I've used it to call some custom Kalman filter code that I had. Calling the Fortran version was significantly faster than the same algorithm translated into numpy, probably due to a bunch of array slicing. However, as numba has gotten better, I've leaned more towards decorating the numpy code with @njit instead.

What is f2py used for while building numpy source?

2 Answers2

Update