DLL file loaded twice with DLL redirection through manifest

Question

I'm including python.h in my Visual C++ DLL file project which causes an implicit linking with python25.dll. However, I want to load a specific python25.dll (several can be present on the computer), so I created a very simple manifest file named test.manifest:

<?xml version='1.0' encoding='UTF-8' standalone='yes'?>
<assembly xmlns='urn:schemas-microsoft-com:asm.v1' manifestVersion='1.0'>
    <file name="python25.dll" />
</assembly>

And I'm merging it with the automatically embedded manifest file generated by Visual Studio thanks to:

Configuration Properties -> Manifest Tool -> Input and Output -> Additional Manifest Files
-->$(ProjectDir)\src\test.manifest

python25.dll is now loaded twice: the one requested by the manifest, and the one that Windows should find through its search order.

Screendump of Process Explorer http://dl.dropbox.com/u/3545118/python25_dll.png

Why is that happening and how can I just load the DLL file pointed by the manifest?

Answer 1

After exhaustive battle with WinSxS and DLL redirection, here's my advice for you:

Some background

Various things can cause a DLL file to be loaded under Windows:

• Explicit linking (LoadLibrary) -- the loader uses the current activation context of the running EXE file. This is intuitive.
• Implicit linking ("load time linkage", the "auto" ones) -- the loader uses the default activation context of the depending DLL file. If A.exe depends on B.dll depends on C.dll (all implicit linkage), the loader will use B.dll's activation context when loading C.dll. IIRC, it means if B's DllMain loads C.dll, it can be using B.dll's activation context -- most of the time it means the system-wide default activation context. So you get your Python DLL from %SystemRoot%.
• COM (CoCreateInstance) -- this is the nasty one. Extremely subtle. It turns out the loader can look up the full path of a DLL file from the registry using COM (under HKCR\CLSID). LoadLibrary will not do any searching if the user gives it a full path, so the activation context can't affect the DLL file resolution. Those can be redirected with the comClass element and friends, see [reference][msdn_assembly_ref].
• Even though you have the correct manifest, sometimes someone can still change the activation context at run time using the Activation Context API. If this is the case, there is usually not much you can do about it (see the ultimate solution below); this is just here for completeness. If you want to find out who is messing with the activation context, WinDbg bp kernel32!ActivateActCtx.

Now on to finding the culprit

1. The easiest way to find out what causes a DLL file to load is to use Process Monitor. You can watch for "Path containing python25.dll" or "Detail containing python25.dll" (for COM lookups). Double clicking an entry will actually show you a stack trace (you need to set the symbol search paths first, and also set Microsoft's PDB server). This should be enough for most of your needs.
2. Sometimes the stack trace obtained from above could be spawned from a new thread. For this purpose you need WinDbg. That can be another topic, but suffice to say you can sxe ld python25 and look at what other threads are doing (!findstack MyExeModuleName or ~*k) that causes a DLL file to load.

Real world solution

Instead of fiddling with this WinSxS thing, try hooking LoadLibraryW using Mhook or EasyHook. You can just totally replace that call with your custom logic. You can finish this before lunch and find the meaning of life again.

[msdn_assembly_ref]: Assembly Manifests

Answer 2

I made some progress for the understanding of the issue.

First let me clarify the scenario:

• I'm building a DLL file that both embeds and extends Python, using the Python C API and Boost.Python.
• Thus, I'm providing a python25.dll in the same folder as my DLL file, as well as a boost_python-vc90-mt-1_39.dll.
• Then I have an EXE file which is a demo to show how to link to my DLL file: this EXE file doesn't have to be in the same folder as my DLL file, as long as the DLL file can be found in the PATH (I'm assuming that the end user may or may not put it in the same folder).

Then, when running the EXE file, the current directory is not the one containing python25.dll, and that's why the search order is used and some other python25.dll can be found before mine.

Now I figured out that the manifest technique was the good approach: I managed to redirect the loading to "my" python25.dll.

The problem is that this is the Boost DLL file boost_python-vc90-mt-1_39.dll that's responsible for the "double" loading!

If I don't load this one, then python25.dll is correctly redirected. Now I somehow have to figure out how to tell the Boost DLL file not to load another python25.dll...

Answer 3

Dependency Walker is usually the best tool for resolving this kind of problem. I'm not too sure how well it handles manifests though...

Where in this entangled mess is the actual process executable file?

Two possibilities come to mind:

1. You are writing a Python extension DLL file. So the Python process is loading your DLL file, and it would already have its own python25.dll dependency.

2. The EXE file loading your DLL file is being built with header files and libraries provided by the DLL file project. So it is inheriting the #pragma comment(lib,"python25.lib") from your header file and as a result is loading the DLL file itself.

My problem with the second scenario is, I'd expect the EXE file, and your DLL file, to be in the same folder in the case that the EXE file is implicitly loading your DLL file. In which case the EXE file, your DLL file and the python25.dll are all already in the same folder. Why then would the system32 version ever be loaded? The search order for implicitly loaded DLL files is always in the application EXE file's folder.

So the actual interesting question implicit in your query is: How is the system32 python26.dll being loaded at all?

Answer 4

Recently, I hit a very similar problem:

1. My application embedding Python loads the python32.dll from a known location, that is a side-by-side assembly (WinSxS) with Python.manifest
2. Attempt to import tkinter inside the embedded Python interpreter caused second loading of the very same python32.dll, but under a different non-default address.
3. The initialisation function of tkinter module (specifically, _tkinter.pyd) was failing because to invalid Python interpreter thread state (_PyThreadState_Current == NULL). Obviously, Py_Initialize() was never called for the second Python interpreter loaded from the duplicate python32.dll.

Why was the python32.dll loaded twice? As I explained in my post on python-capi, this was caused by the fact the application was loading python32.dll from WinSxS, but _tkinter.pyd did not recognise the assembly, so python32.dll was loaded using the regular DLL search path.

The Python.manifest + python32.dll assembly was recognised by the DLL loading machinery as a different module (under different activation context), than the python32.dll requested by _tkinter.pyd.

Removing the reference to Python.manifest from the application embedding Python and allowing the DLL search path to look for the DLLs solved the problem.