ImportError when using IPython only

Question

I have a compiled .pyd module that relies on several other DLLs (Python mapscript and MapServer) and I'm trying to use it in IPython on Windows 10. Set up steps are as follows:

• Created a new virtual environment
• installed mapscript to the environment
• set the PATH to include the folder containing the MapServer DLLs
• the "import mapscript" works fine when run using the standard venv Python
• when starting ipython and running "import mapscript" I get the following error: ImportError: DLL load failed: The specified procedure could not be found.

Debugging steps so far:

• sys.executable is identical in both cases (venv Python and venv IPython)
• os.environ["PATH"] is identical
• sys.path in ipython has a couple of extra paths in IPython at the end - C:\Users\user\.ipython and 'c:\virtualenvs\mapscript-jupyter3\lib\site-packages\IPython\extensions' but are otherwise identical. Removing these paths does not change the error.
• I've tried finding a missing DLL using Process Monitor but they all seem to be found.

The same issue occurs in both Python 2.7 and Python 3.6 with newly created venvs. The issue seems to be the opposite of ipython notebook can import a pyd module but the python interpter can't

So my question is what does IPython do to the Python environment that can cause differences to standard Python and cause the ImportError?

Answer 1

This took quite some time to find the issue. Any missing DLLs are quite easy to spot by using Process Monitor. In this case however all the dependent DLLs were found, but one of them didn't have a function used by the .pyd file (or associated DLLs) I was trying to import.

I managed to narrow down the test to 2 commands that caused the error - loading the PYD directly.

python -c "import _mapscript"
ipython -c "import _mapscript"

I tried removing / modifying the PATH, in and out of virtual environments, and py2 and py3 and all produced the same error.

Then I tried running Process Monitor and comparing results - nothing stood out apart from ipython loads lots of additional Python libraries for the interactive shell.

Process Monitor includes a handy Properties option for each event, which also shows which modules are loaded for a Process. These can be sorted and then copied to the clipboard.

I was able to compare the output of the working Python and broken IPython processes. IPython included several additional .pyd files from the root Python installation (C:\Python36\DLLs). I knew the pyd I was trying to load used sqlite, and this was one of the loaded modules (presumably as IPython stores all input commands in a sqlite database to easily access the history). Temporarily removing the _sqlite3.pyd file allowed the module to be loaded.

The Python DLLs folder takes priority over folders on the PATH, so the current fix was to replace the sqlite3.dll in the Python DLL folder with the one used by MapServer and all worked correctly.