Adding output file to Python extension

Question

I've defined a custom build_ext to build a funky extension that I'm trying to make pip-friendly. The following is a trimmed version of what I'm doing.

foo_ext = Extension(
  name='foo/_foo',
  sources=foo_sources,
)

class MyBuildExt(build_ext):
  def build_extension(self, ext):
    # This standalone script builds the __init__.py file 
    #  and some .h files for the extension
    check_call(['python', 'build_init_file.py'])

    # Now that we've created all the init and .h code
    #  build the C/C++ extension using setuptools/distutils
    build_ext.build_extension(self, ext)

    # Include the generated __init__.py in the build directory 
    #  which is something like `build/lib.linux-x86/foo/`.  
    #  How can I get setuptools/distutils to install the 
    #  generated file automatically?!
    generated_file = 'Source/foo/__init__.py'
    output_path = '/'.join(self.get_outputs()[0].split('/')[:-1])
    self.move_file(generated_file, output_path)

setup(
    ...,
    ext_modules = [foo_ext],
    cmdclass={'build_ext' : MyBuildExt},
)

After packaging this module up and installing it with pip, I have a module foo in my virtualenv's site-packages directory. The directory structure looks like the following.

foo/
foo/__init__.py
foo/_foo.so

The egg-info/SOURCES.txt file does not include the __init__.py file that I manually created/moved. When I do a pip uninstall foo the command leaves foo/__init__.py in my virtualenv's site-packages. I would like pip to delete the entire package. How can I add the generated __init__.py file that I manually move into the build directory to the list of output files that are installed?

I realize this is disgusting and hacky, so I welcome disgusting and hacky answers!

Attempts:

Added packages=['foo'] -- When I do, pip doesn't build extension. Also tried adjusting file path/namespace versions of the package name -- no difference.

score 5 · Accepted Answer · answered May 16 '12 at 04:34

In order to have distutils install a Python package, you need to pass packages=['foo'], and if you’re putting it somewhere that is not the root level of the project (I mean the foo directory next to the setup.py script), like you seem to do here, you also have to pass package_dir={'foo': 'Source'} or to use a simpler layout. If your setup.py script contains this packages parameter then the build command will invoke the build_py command to move the Python source files (and directories) to the build directory, which will be copied later by the install command.

The problem here is that your foo/__init__.py file is built by the build_ext command, which runs after build_py. You need to override that with a custom build command:

class MyBuild(build):
  sub_commands = [('build_clib', build.has_c_libraries),
                  ('build_ext', build.has_ext_modules),
                  ('build_py', build.has_pure_modules),
                  ('build_scripts', build.has_scripts),
                 ]

setup(..., cmdclass={'build': MyBuild, 'build_ext': MyBuildExt})

The elements in the sub_commands attribute are tuples of (command name, function to call to decide whether to run the command or not); this is documented in the source code but I don’t remember if it’s explained in the doc. In the standard build class build_py precedes build_clib. I may change this in the next version of Python 2.7, as it was reported that it interacted badly with 2to3 conversion.

This worked perfectly and I learned something about distutils. Thanks, Éric! — sholsapp, May 16 '12 at 15:49

score 1 · Answer 2 · answered May 12 '12 at 03:56

1

First, the name parameter in your Extension instance should be a module name (foo._foo), not a path.

Have you tried adding packages=['foo'] to your setup call?

answered May 12 '12 at 03:56

merwok

6,779
1
28
42

I think the file hierarchy/namespace difference translates to the same thing. – sholsapp May 12 '12 at 06:01
Still, it doesn’t hurt to use the right format instead of relying on it working by luck. I think I have found how to fix your problem, I’ll post another answer tomorrow if I find the time. Keep hope :) – merwok May 15 '12 at 04:55

Adding output file to Python extension

Attempts:

2 Answers2