Why can't the embeddable Python distribution locate native modules?

Question

I'm trying to build a standalone distribution of WeasyPrint for Windows. I downloaded and unpacked the embeddable Python 3.8 ZIP file to a directory called dist. Then, in MSYS2, I used a full Python 3.8 interpreter with Pip to install WeasyPrint into the dist directory (accounting for a bug in Pillow when installing on MSYS2):

pip install --target dist --global-option=build_ext --global-option="-ldl" pillow
pip install --target dist weasyprint

Now, when I run WeasyPrint:

./dist/python.exe -m weasyprint http://weasyprint.org test.pdf

I get:

Traceback (most recent call last):
  File "runpy.py", line 184, in _run_module_as_main
  File "runpy.py", line 143, in _get_module_details
  File "runpy.py", line 110, in _get_module_details
  File "C:\msys64\home\David\dist\weasyprint\__init__.py", line 440, in <module>
    from .css import preprocess_stylesheet  # noqa isort:skip
  File "C:\msys64\home\David\dist\weasyprint\css\__init__.py", line 30, in <module>
    from . import computed_values, media_queries
  File "C:\msys64\home\David\dist\weasyprint\css\computed_values.py", line 18, in <module>
    from .. import text
  File "C:\msys64\home\David\dist\weasyprint\text.py", line 14, in <module>
    import cairocffi as cairo
  File "C:\msys64\home\David\dist\cairocffi\__init__.py", line 17, in <module>
    from ._generated.ffi import ffi
  File "C:\msys64\home\David\dist\cairocffi\_generated\ffi.py", line 2, in <module>
    import _cffi_backend
ModuleNotFoundError: No module named '_cffi_backend'

How is Python able to find the other modules, but not the native one? There's a file named _cffi_backend-cpython-38.dll in the dist directory. It still doesn't work if I rename the file to just _cffi_backend.dll.

I'm not sure python can read `.dll` file as module. Try to rename it as `.pyd` — Smankusors, Mar 08 '20 at 08:02

score 0 · Answer 1 · answered Mar 08 '20 at 12:10

As Smankusors alluded to in a comment, this is caused by MSYS2 Python and embeddable Python using different suffixes for extension modules:

Embeddable Python:

$ ./dist/python.exe -c 'from importlib.machinery import EXTENSION_SUFFIXES; print(EXTENSION_SUFFIXES)'
['.cp38-win_amd64.pyd', '.pyd']

MSYS2 Python:

$ python -c 'from importlib.machinery import EXTENSION_SUFFIXES; print(EXTENSION_SUFFIXES)'
['-cpython-38.dll', '-abi3.dll', '.cp38.pyd', '.pyd']

So when the CFFI wheel is built with MSYS2 Python, the MSYS2 Python suffix is used, but Embeddable Python doesn't know to look for it. The fix is to rename the DLLs to use one of the Embeddable Python suffixes or use a native Windows installation of Python to install modules.

Why can't the embeddable Python distribution locate native modules?

1 Answers1