Using embedded python, loading a *.pyd outside of DLLs directory fails

Question

I have a C++ application (X-Plane) for which there is a plugin which permits the use of python scripts (XPPython3 plugin). The plugin is written in C, using python CAPI, and works great, allowing me to write python scripts which get executed within the C++ application.

On Windows 10, I want to extend my python features by importing imgui. I have a python cython-built pyd file (_imgui.cp39-win_amd64.pyd).

• If I place the pyd file in C\Program Files\Python39\DLLs, it works as expected: C++ application calls CAPI to python, which loads script which imports and executes imgui code.

• If I place the pyd file anywhere else, embedded python either reports "module not found" -- if the pyd isn't on sys.path(), or if it is on sys.path():

  ImportError: DLL load failed while importing _imgui: The parameter is incorrect.'

Changes using: os.add_dll_directory(r'D:\somewhere\else') Does not effect whether the module is found or not, nor does it change the 'parameter incorrect' error. (see https://bugs.python.org/issue36085 for details on this change. -- my guess is add_dll_directory changes lookup for DLLs, but not for pyd?) sys.path appears to be used for locating pyd.

Yes, the pyd is compiled with python3.9: I've compiled it both with mingw and with visual studio toolchains, in case that might be a difference.

For fun, I moved python-standard _zoneinfo.pyd from Python39\DLLs and it fails in the same way in embedded python: "The parameter is incorrect". So, that would appear to rule out my specific pyd file.

The key question is/are:

• Other than placing a pyd file under PythonXX\DLLs, is there a way to load a PYD in an embedded python implementation? (I want to avoid having to tell users to move my pyd file into the Python39\DLLs directory... because they'll forget.)

Note that using IDLE or python.exe, I can load pyds without error -- anywhere on sys.path -- so they don't have to be under Python39\DLLs. It's only when trying to load from embedded python that the "Parameter is incorrect" appears. And of course, this works flawlessly on Mac.

(Bonus question: what parameter? It appears to be python passing through a windows error.)

Answer 1

There seems to be a simple answer, though I suspect it's better characterized as a python bug.

There is nothing magical about Python39\DLLs directory.

The problem is using absolute vs relative paths in sys.path.

Python can find modules using absolute or relative paths. So if zippy.py is in folder foobar,

sys.path.append('foobar')
import zippy
# Success

Python and find, BUT NOT LOAD pyd files using relative paths. For example, move _zoneinfo.pyd from PythonXX\LDDs to foobar

sys.path.append('foobar')
import _zoneinfo
# ImportError: DLL load failed while importing _zoneinfo: The parameter is incorrect.'

Instead, use absolute path, and it will find and load PYD:

sys.path.append(r'c:\MyTest\foobar')
import _zoneinfo
# Success

Answer 2

So, there is actually a way to do this—that is, ship your application with the desired libraries. The solution is to use an embedded distribution and ship this with your application. You can find the correct distribution on the official Python download page corresponding to your desired version (here's the link to the lastest 3.9 release which seems to be what you're using: https://www.python.org/downloads/release/python-392/). Look for the Windows Embeddable Package.

You can then simply drop in your .pyd file alongside the standard library files (note that if your third-party library is dependent on any other libraries, you will have to include them, as well). Shipping your application with an embeddable distribution should not only solve your current issue, but will also mean that your application will work regardless of which version of Python a user has installed (or without having Python installed at all).