Py2app and Py2exe fail to package Patsy files

Question

I have been building a python-based standalone application using py2app (Mac OS X 10.6) and py2exe (Win XP and Win7). Recently I added support for functions that depend on the library patsy. However, when building py2app or py2exe versions of my software, only the "init.pyc" and "origin.pyc" files from patsy are included in the site-packages.zip patsy folder (excluding the 20 additional python module files). When importing patsy, the standalone app throws the error "ImportError: No module named highlevel" (highlevel is patsy module).

An ugly work around for this is to throw the missing .pyd modules into the patsy directory of the site-packages.zip, but this is not ideal. A simple example of this for py2app code is shown, but the same applies for py2exe:

test_import.py

import patsy
print 'hello patsy'

setup.py

includes = ["patsy"]

from distutils.core import setup
import py2app
import patsy
from patsy import highlevel

options = {"py2app":{"includes": includes}}
setup(name='test',app=["test_import.py"],setup_requires=["py2app"])

Any recommendations are greatly appreciated. Thanks.

Answer 1

The problem is that patsy is slightly over-clever in how it defines its main namespace, see the end of patsy/__init__.py:

def _reexport(modname):
    __import__(modname)
    mod = sys.modules[modname]
    for var in mod.__all__:
        __all__.append(var)
        globals()[var] = getattr(mod, var)
for child in ["highlevel", "build", "constraint", "contrasts",
              "desc", "design_info", "eval", "origin", "state",
              "user_util", "missing", "splines"]:
    _reexport("patsy." + child)

This avoids a bunch of copy/paste code duplication but messes up py2exe and py2app's heuristics for figuring out which files are in use; because there's no explicit literal import statement, they can't "see" that all those modules are in fact getting imported. The real solution is for py2exe/py2app to catch up with the last 5 years of python infrastructure development and trust packages' setup.py to install only what's needed.

In the mean time, this is a very standard problem so both packages have ways to work around it, e.g. for py2app these appear to be called "recipes": http://pythonhosted.org/py2app/recipes.html

Edit: Alternate solution: use the just-released patsy 0.2.1, which has a slightly less clever __init__.py which should Just Work.