Deploy Python programs on Windows and fetch big library dependencies

Question

I have some small Python programs which depend on several big libraries, such as:

• NumPy & SciPy
• matplotlib
• PyQt
• OpenCV
• PIL

I'd like to make it easier to install these programs for Windows users. Currently I have two options:

• either create huge executable bundles with PyInstaller, py2exe or similar tool,
• or write step-by-step manual installation instructions.

Executable bundles are way too big. I always feel like there is some magic happening, which may or may not work the next time I use a different library or a new library version. I dislike wasted space too. Manual installation is too easy to do wrong, there are too many steps: download this particular interpreter version, download numpy, scipy, pyqt, pil binaries, make sure they all are built for the same python version and the same platform, install one after another, download and unpack OpenCV, copy its .pyd file deep inside Python installation, setup environment variables and file asssociations... You see, few users will have the patience and self-confidence to do all this.

What I'd like to do: distribute only a small Python source and, probably, an installation script, which fetches and installs all the missing dependencies (correct versions, correct platform, installs them in the right order). That's a trivial task with any Linux package manager, but I just don't know which tools can accomplish it on Windows. Are there simple tools which can generate Windows installers from a list of URLs of dependencies¹?

¹ As you may have noticed, most of the libraries I listed are not installable with pip/easy_install, but require to run their own installers and modify some files and environment variables.

Answer 1

npackd exists http://code.google.com/p/windows-package-manager/ It could be done through here or use distribute (python 3.x) or setuptools (python 2.x) with easy_install, possibly pip (don't know it's windows compatibility). But I would choose npackd because PyQt and it's unusual setup for pip/easy_install (doesn't play with them nicely, using a configure.py instead of setup.py). Though you would have to create your own repo for npackd to use for some of them. I forget what is contributed in total for python libs with it.

Answer 2

AFAIK there is no tool (and I'd assume you googled), so you must make one yourself.

Fetching the proper library versions seems simple enough -- using python's ftplib you can fetch the proper installers for every library. How would you know which version is compatible with the user's python? You can store different lists of download URLs, each for a different python version (this method came off the top of my head and there is probably a better way; not that it matters much if it's simple and it works).

After you figure out how to make each installer run, you can py2exe your installer script, and even use it to fetch the program itself.

---

EDIT

Some Considerations

There are a couple of things that popped into my mind just as I posted: First, some pseudocode (how I would approach it, anyway)

#first, we check modules
try:
    import numpy
except ImportError:
    #flag numpy for installation
#lather, rinse repeat for all dependencies

#next we check version compatibility -- note that if a library version you need 
#is not backwards-compatible, you're in DLL hell, and there is little we can do.
<insert version-checking code here>

#once you have your unavailable dependencies, you install them
import ftplib
<all your file-downloading here>

#now you install. sorry I can't help you here.

There are a few things you can do to make your utility reusable --

• put all URL lists, minimum version numbers, required library names etc in config files
• Write a script which helps you set up an installer
• Py2exe the installer-maker-script
• Sell it
• Even better, release it under GPL so we can all feast upon fruits of your labours.

Answer 3

I have a similar need as you, but in addition I need the packaged application to work on several platforms. I'm currently exploring the currently available solutions, here are a few interesting ones:

• Use SnakeBasket, which wraps around Pip and add a recursive dependency resolution plus a heuristic to choose the right version when there are conflicts.
• Package all dependencies as an egg, but not your sourcecode which will still be editable: https://stackoverflow.com/a/528064/1121352
• Package all dependencies in a zip file and directly import the modules on the fly: Cross-platform alternative to py2exe or http://davidf.sjsoft.com/mirrors/mcmillan-inc/install1.html
• Using buildout: http://www.buildout.org/en/latest/install.html
• Using virtualenv with virtualenv-tools (instead of "relocate")
• If your main problem when freezing your code using PyInstaller or similar is that you end up with a big single file, you can customize the process so that you get several files, one for each dependency, instead of one big executable.

I will update here if I find something that fills my bill.