5

I am using Cython as part of my build setup for a large project, driven by CMake. I can't seem to get Cython to generate the .c files in a sensible location.

My file layout:

C:\mypath\src\demo.py                   # Cython source file
C:\mypath\build\bin                     # I want demo.pyd to end up here
C:\mypath\build\projects\cyt\setup.py   # Generated by CMake

My setup.py is generated by CMake (a lot of it depends on configure_file), in the location specified above. This location conforms to the usual structure of the overarching project (which builds over a hundred libraries and executables) and is not something I want to (or can easily) change.

The generated setup.py looks like this:

from distutils.core import setup, Extension
from Cython.Build import cythonize
import os.path

extension_args = {
    'extra_compile_args' : ['/DWIN32','/DWIN64'],
    'extra_link_args'    : ['/MACHINE:X64'],
}

source = '../../../src/demo.py'
modules = [Extension(
    os.path.splitext(os.path.basename(source))[0],
    sources = [source],
    **extension_args
    )]

modules = cythonize(
    modules,
    build_dir = 'BUILD_DIR',
    compiler_directives = {'language_level' : 2}
)

setup(name        = 'demo',
      version     = '0.1',
      description = '',
      ext_modules = modules)

(Note that this is heavily simplified compared to the real case, which passes many additional arguments in extension_args, and includes many source files, each with its own object in modules. Nevertheless, I have verified that the minimised version above reproduces my issue).

Cython is run like this:

cd C:\mypath\build\projects\cyt
python setup.py build_ext --build-lib C:/mypath/build/bin --build-temp C:/mypath/build/projects/cyt

Ideally, I would want all intermediary build artefacts from Cython (the generated C files, object files, exp files, etc.) to reside somewhere in or below C:\mypath\build\projects\cyt. However, I can't seem to achieve that. Here is where build artefacts actually end up:

  • demo.pyd ends up in C:\mypath\build\bin, where I want it. No problem here.
  • The object file demo.obj, along with the linked files demo.exp and demo.lib, end up in C:\mypath\build\projects\src. I want them inside cyt.
  • The C file demo.c ends up in C:\mypath\build\src. Again, I want this in projects\cyt.

In the setup.py, I am setting the build_dir parameter for cythonize as suggested in this answer, but it doesn't seem to work as I would like. I also tried using cython_c_in_temp as per another answer on that question, but that has no effect (and judging from my inspection of Cython source code, does not apply to cythonize calls at all).

I tried using an absolute paths for source, but that made things even worse, as the C file ended up generated right next to demo.py, inside the source tree (as C:\src\demo.c).

My question: How can I make sure that all the generated intermediary files (C, obj, and friends) end up in the same directory as the generated setup.py, or below that?

I can think of two workarounds for my situation, but they both feel like hacks which I would like to avoid:

  1. Copy all the Python source files from their locations in C:\mypath\src to alongside the generated setup.py, so that I can refer to them without .. in the path. That would likely solve the issue, but burdens the (already long) build process with tens of additional file copy operations I'd rather avoid.
  2. Since the path where the files end up seems to be composed by concatenating "the directory of setup.py + the value of build_dir + the value of source", I could count the number of .. in the source path and specify build_dir deep enough so that the evaluation results in the path I actually want. This is both extremely hacky and very fragile.

I hope a better solution exists.

Angew is no longer proud of SO
  • 167,307
  • 17
  • 350
  • 455

1 Answers1

3

So it seems like you've run into a bug. Here's the relevant section of code in Cython. Basically, cythonize will try to build the paths to your .c and .o files as so:

C:/mypath/build/projects/cyt/BUILD_DIR/../../../src/demo.c

and so instead of nicely contained temporary files, you end up with insanity. Using an absolute path to demo.py won't help either, since that same code will just pass absolute paths through unchanged.

There doesn't seem to be a way to fix this in user-space short of extensive monkey-patching, so I submitted a pull-request to Cython with an actual fix. Once that's been merged in, you should then be able to run:

cd C:\mypath\build\projects\cyt
python setup.py build_ext -b C:/mypath/build/bin -t .

to get the result you want (-b and -t are the short forms of --build-lib and --build-temp).

tel
  • 13,005
  • 2
  • 44
  • 62