I have a Python package that contains Cython code, similar to scikit-learn. I am trying to use poetry to do package management, but meson to handle the build process for Cython and possible c++ code. Before, when I was using the deprecated setuptools and distutils, I was able to do the following in scikit-learn:
pip install -e .
which built the Cython files and allowed them to be importable by sklearn's Python files. For example from _criterion import Criterion works, where _criterion.pxd/pyx is a Cython file that contains some variable named Criterion.
Now I am using poetry and meson and things do not work.
My Goals to Use Poetry and Meson
I would ideally like to be able to just do poetry install (or poetry build and poetry install), which should build and install things by using Meson to build local Cython/C++ files and then use poetry to handle other pip installable Python packages.
My Question
How do I correctly use poetry with meson to build my Cython files and install them along with other relevant Python dependencies to install my package?
My Attempt using just Meson incorrectly Cython files to the wrong environment
I instantiated my meson.build files such that running meson build in my repo's root directory works:
The Meson build system
Version: 0.63.2
Source dir: /Users/adam2392/Documents/scikit-tree
Build dir: /Users/adam2392/Documents/scikit-tree/build
Build type: native build
Project name: scikit-tree
Project version: 0.0.0.dev0
C compiler for the host machine: arm64-apple-darwin20.0.0-clang (clang 14.0.6 "clang version 14.0.6")
C linker for the host machine: arm64-apple-darwin20.0.0-clang ld64 609
C++ compiler for the host machine: arm64-apple-darwin20.0.0-clang++ (clang 14.0.6 "clang version 14.0.6")
C++ linker for the host machine: arm64-apple-darwin20.0.0-clang++ ld64 609
Host machine cpu family: aarch64
Host machine cpu: arm64
Library m found: YES
Program cython found: YES (/Users/adam2392/miniforge3/envs/sktree/bin/cython)
Program python3 found: YES (/Users/adam2392/miniforge3/envs/sktree/bin/python3)
Message: /Users/adam2392/miniforge3/envs/sktree/bin/python3
Message: /opt/homebrew/lib/python3.9/site-packages/
Found pkg-config: /opt/homebrew/bin/pkg-config (0.29.2)
Program _build_utils/cythoner.py found: YES (python /Users/adam2392/Documents/scikit-tree/sktree/_build_utils/cythoner.py)
Build targets in project: 1
scikit-tree 0.0.0.dev0
User defined options
backend: ninja
Found ninja-1.11.1 at /opt/homebrew/bin/ninja
When I then run meson install within the build/ directory:
(sktree) adam2392@Adams-MacBook-Pro build % meson install
ninja: Entering directory `/Users/adam2392/Documents/scikit-tree/build'
[3/3] Linking target sktree/tree/_unsup_criterion.cpython-39-darwin.so
ld: warning: -pie being ignored. It is only used when linking a main executable
Installing sktree/tree/_unsup_criterion.cpython-39-darwin.so to /opt/homebrew/lib/python3.9/site-packages/sktree/tree
Installing /Users/adam2392/Documents/scikit-tree/sktree/__init__.py to /opt/homebrew/lib/python3.9/site-packages/sktree
Installing /Users/adam2392/Documents/scikit-tree/sktree/_forest.py to /opt/homebrew/lib/python3.9/site-packages/sktree
Installing /Users/adam2392/Documents/scikit-tree/sktree/tree/__init__.py to /opt/homebrew/lib/python3.9/site-packages/sktree/tree
Installing /Users/adam2392/Documents/scikit-tree/sktree/tree/_classes.py to /opt/homebrew/lib/python3.9/site-packages/sktree/tree
which installs for example the Cython files into the incorrect Python environment (i.e. home-brew). I need it to be installed in my virtual environment.
My Attempt using Poetry doesn't work either
My pyproject.toml file has the following lines:
[build-system]
build-backend = "mesonpy"
requires = [
"meson-python>=0.11.0",
# `wheel` is needed for non-isolated builds, given that `meson-python`
# doesn't list it as a runtime requirement (at least in 0.10.0)
# See https://github.com/FFY00/meson-python/blob/main/pyproject.toml#L4
"wheel",
"setuptools<=65.5",
"packaging",
"Cython>=0.29.24",
"scikit-learn",
]
However, when I run poetry install, there are no errors, but:
python -c "from sktree import tree"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/adam2392/Documents/scikit-tree/sktree/__init__.py", line 37, in <module>
from . import tree
File "/Users/adam2392/Documents/scikit-tree/sktree/tree/__init__.py", line 1, in <module>
from ._classes import UnsupervisedDecisionTree
File "/Users/adam2392/Documents/scikit-tree/sktree/tree/_classes.py", line 3, in <module>
from ._unsup_criterion import UnsupervisedCriterion
ModuleNotFoundError: No module named 'sktree.tree._unsup_criterion'
