2

I'm trying to embed python in a c/c++ application, using the embedded python zip file. I also want to use site-packages, like numpy. I manipulate the sys path myself, so it'll include all paths necessary.

What I have found out for now is:

  1. If I use Py_Initialize/Py_Finalize more than once, my application will crash the 2nd time I run a script when using numpy, due to numpy being imported twice during the same process and Py_Finalize doesn't clean up the mess numpy leaves behind.

  2. If I use sub interpreters, the same happens, because, again, numpy gets imported each time "import" is called in a sup interpreter.

  3. If I use a single interpreter (calling Py_Initialize at application start and then calling Py_Finalize at application shutdown), Py_RunFile() will run smoothly as long as I do not use site-packages. If I do import a site-package (e.g. import numpy) Py_RunFile will hang, though. The application will work, because the python interpreter is running in its own thread, but the interpreter itself will hang.

I know that the site-package *.pyd files are loaded:

'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\DLLs\_ctypes.pyd'. Symbols loaded.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\Lib\site-packages\numpy\.libs\libopenblas.TXA6YQSD3GCQQC22GEQ54J2UDCXDXHWN.gfortran-win_amd64.dll'. Module was built without symbols.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\Lib\site-packages\numpy\core\_multiarray_umath.cp37-win_amd64.pyd'. Module was built without symbols.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\Lib\site-packages\numpy\core\_multiarray_tests.cp37-win_amd64.pyd'. Module was built without symbols.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\Lib\site-packages\numpy\linalg\lapack_lite.cp37-win_amd64.pyd'. Module was built without symbols.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\Lib\site-packages\numpy\linalg\_umath_linalg.cp37-win_amd64.pyd'. Module was built without symbols.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\DLLs\_bz2.pyd'. Symbols loaded.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\DLLs\_lzma.pyd'. Symbols loaded.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\DLLs\_decimal.pyd'. Symbols loaded.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\Lib\site-packages\numpy\fft\fftpack_lite.cp37-win_amd64.pyd'. Module was built without symbols.
'xxx.exe' (Win32): Loaded 'C:\tools\python37x64\Lib\site-packages\numpy\random\mtrand.cp37-win_amd64.pyd'. Module was built without symbols.

but then nothing happens.

So, can anybody tell me what I'm doing wrong that my interpreter hangs when I try to use site-packages?

edit:

I don't call any other python specific functions, other than those to manipulate sys.path and Py_Initialize

edit 2:

I have found this discussion.

But that guy had the problem that the program would hang if he used sub interpreters. But using sub interpreters for me works once and then it crashes (as mentioned above). It seems, though, as if the same line is leading to a deadlock here:

python37.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv, _RTL_CRITICAL_SECTION * cs, unsigned long ms) Line 187 C
python37.dll!take_gil(_ts * tstate) Line 208    C
python37.dll!PyEval_RestoreThread(_ts * tstate) Line 273    C
python37.dll!PyGILState_Ensure() Line 1067  C
_multiarray_umath.cp37-win_amd64.pyd!00007ffdab6ee940() Unknown
Jan
  • 2,853
  • 2
  • 21
  • 26
wickermoon
  • 171
  • 1
  • 10
  • I'm having this same issue... were you ever able to get past it? – textral Feb 06 '20 at 07:30
  • I think with python 3.8 a lot has been done so that when loading a dll fails, python doesn't just drag the whole process down. We switched to an IPC solution, though, where we call python from our process, but in a new process, communicating via named pipes and shared memory. – wickermoon Feb 06 '20 at 08:31
  • Yes, I'm also considering a separate process, or simply using system() to invoke the python bits. I wonder if the problem is that the numpy DLLs are ABI-incompatible with my MSVC-compiled project? I think numpy is typically built with GNU compilers – textral Feb 06 '20 at 23:20

0 Answers0