8

I can load Python modules (.py, .pyc, .pyd) from a zip file by calling "import some_module" from a Python interpreter only after sys.path has been extended to include the zip file and only after I have run

import zipextimporter
zipextimporter.install()

The latter is required for .pyd modules.

I can also load Python .py and .pyc modules from Python embedded in C++. However, in order to also load .pyd modules from embedded Python I added

PyRun_SimpleString("import zipextimporter");

The C++ exe runs beyond this line without error. But the next command

PyRun_SimpleString("zipextimporter.install()");

gives me this error:

enter image description here

Why does zipextimporter.install() crash when Python is embedded?

How can I solve this?

Does it perhaps have to do with the way the C++ code is compiled? I use g++:

g++ embed-simple.cpp -IE:\Python27\include -LE:\Python27\libs -lpython27 -o embed-simple

I saw a link How to link against msvcr90.dll with mingw gcc?

Could that provide a solution? If yes, how should I adjust it, gcc-->g++, since I am running C++ code, not C.

I am running Python 2.7.2 on WinXP.

I don't get the runtime error after a clean install of Python 2.7.2, just this:

Import Error: No module named....

Would it matter the way the embedding C++ script is compiled? I used g++. I also compiled with the Intel compiler, but that gave the same runtime error. Perhaps I should try MS Visual C++.

Or use ctypes to import the pyd?

Community
  • 1
  • 1
Alex van Houten
  • 308
  • 3
  • 17
  • I've also stumbled upon this one... I'm trying to import PyCrypto and when i use the embedded version i get the same error. Bountying this. – Chiguireitor Mar 13 '12 at 19:34

4 Answers4

6

memimporter and zipextimporter are indeed able to load .pyd files from memory/zip-archives without unpacking them to files.

The runtimerror R6034 is caused by the fact that the VC9 runtime library must be loaded via a manifest. Running your code in Python 2.5, whic uses a different C runtime, would most probably succeed. I guess you must embed a manifest referencing the VC9 runtime library in your exe; maybe the py2exe wiki can provide some guidance.

theller
  • 2,809
  • 19
  • 19
  • Thanks. Does this mean that if I compile the embedding C++ code with Visual Studio, it will probably work, since Visual Studio by default tries to embed the manifest? – Alex van Houten Jan 03 '12 at 09:12
  • 1
    I dont know (I don't write C/C++ programs anymore ;-) – theller Jan 03 '12 at 16:32
  • I am still confused. If zipextimporter.pyc is inside the zip, _memimporter will also be inside the zip. But the latter is a .pyd file which we can't load, since we are searching for a tool which enables us to do just that: loading pyd files from within a zip. – Alex van Houten Jan 18 '12 at 16:43
  • Sure, _memimporter.pyd cannot be loaded from a zip file. That's why py2exe has it as builtin module. – theller Feb 09 '12 at 11:10
  • But then how can you get your Python code (which calls a few Cython modules) working on a machine without Python, if you want to supply the users of that machine with a single zip file for reasons of convenience? – Alex van Houten Feb 10 '12 at 09:15
  • You can pack a module into a resource in your EXE, then unpack it into a temporary file by hand. Or use any other non-python way (e.g. zlib) – ivan_pozdeev Mar 20 '12 at 18:15
4

PYD files are native DLL files with renamed extension. Loading them relies on operating system facilities and operating system restrictions.

Windows XP, or any OS as far as I know, cannot load DLL files from ZIP files, thus you cannot load PYD files from ZIP files.

Mikko Ohtamaa
  • 82,057
  • 50
  • 264
  • 435
  • 1
    Your answer seems incorrect. It does work from a Python interpreter on WinXP, it does not from embedded Python on WinXP. – Alex van Houten Dec 22 '11 at 09:44
  • Hmmm. I must be wrong then :( But that's probably the cause of the issues... DLL loading honours some OS specific rules, like loading DLL files from the same directory as EXE. They might not be laoded from ZIP at all, but it just "happens to work for some reason". Instead of modifying sys.path try modify os.environ["PATH"] which might give Windows some hints where to load DLLs from. – Mikko Ohtamaa Dec 24 '11 at 12:27
  • No. Checked that. It only works from a Python interpreter after I adjusted sys.path to include the zip. – Alex van Houten Dec 24 '11 at 21:00
3

Which version of python was memimporter.pyd (which is inside the zipextimporter) compiled for? If the python interpreter and pyd don't match you're going to get horrible crashes.

I'm not sure where the mem importer code is, but at a guess I would think that the idea is to insert an import hook which detects a zip-based pyd import and extracts the pyd to a temporary location and call the Python interpreter's standard import on that.

Tom Whittock
  • 4,081
  • 19
  • 24
  • I think this is the code: http://src.chromium.org/svn/trunk/tools/third_party/python_26/Lib/site-packages/zipextimporter.py but I am bot sure which version of Python memimporter was compiled for. How do I find that out? – Alex van Houten Dec 22 '11 at 16:03
  • where did you download it from? they usually tell you on or near the download link. The link you pointed me at has python 2.6 in its path, for example – Tom Whittock Dec 22 '11 at 16:12
  • But one can run "import _memimporter" from any Python interpreter. However, it might take the zipextimporter.pyc from inside the zip, which was generated by py2exe in my case. I tried adjusting sys.path to put the zip at the end instead of the beginning, i.e., PyRun_SimpleString("sys.path.insert(-1,\".\\python27.zip\")"); instead of PyRun_SimpleString("sys.path.insert(0,\".\\python27.zip\")"); but that did not make a difference. – Alex van Houten Dec 22 '11 at 16:19
  • memimporter is part of the standard Python installation, I guess. At least 2.7.2 has it. – Alex van Houten Dec 22 '11 at 16:26
  • 1
    It's not a standard module as far as I can tell - it probably got installed into your site packages by py2exe. Hopefully you installed a py2exe which matches your version of the python interpreter? And the python install you have on your machine matches the interpreter you're embedding in your program? – Tom Whittock Dec 22 '11 at 16:57
  • Sorry, I was mistaken, memimporter comes with py2exe. Yes, I did install py2exe for Python 2.7. I only have one Python install on my machine, so that should be ok. – Alex van Houten Dec 22 '11 at 19:36
  • I tried again on a different computer with a fresh install of WinXP, Python 2.7.2 and Py2exe for Python 2.7. I do NOT get the runtime error, just "Import Error: No module named some_module". It cannot find the .pyd. I also installed PyWin (Python for Windows extensions), but that did not make a difference. – Alex van Houten Dec 24 '11 at 21:05
  • How did you compile for xp? I think the answer by theller is more on track. – Tom Whittock Jan 10 '12 at 08:58
2

this sounds like the same kind of issues I had trying to compile an app with py2exe. see here: http://www.py2exe.org/index.cgi/Tutorial, look at section 5.2. Same thing happens... first time you try to use memimporter, it crashes with a similar error message. Basically, for python 2.6+ you need to have the exact version of the runtime library in the path, and a manifest that points to it. Since you are using embedded python, i don't know how it all works, but maybe that will get you closer.

i'd start by putting the 'correct' version of the runtime dll somewhere, and in your python code, before importing zipextimporter, append the path of the correct runtime to sys.path and see if that fixes it. not sure how you get the manifest in there for an embedded python though. it might need to be included in the parent app's manifest.

edit: BTW, i forgot, we found another way around this issue as well. I forget the exact details, but what happens is that your app loads the default version of the runtime, and then python asks for its version, and it finds one in c:\python{26,27} and it doesn't match. The simplest way to solve this problem is to delete c:\python\msvcr90.dll. Then, python won't hit the local (old) version of the dll which might not work with your app's manifest, and both of them will have to go out and get the current version from the windows directory, which will match.

Corley Brigman
  • 11,633
  • 5
  • 33
  • 40