How to prevent a specific DLL from loading into my process

Question

I think I have researched this pretty thoroughly and I have not found an acceptable answer. First the broad strokes: Windows 8.1, Visual Studio 2013. Although, I don't think these are important.

Problem as follows. The application I am writing makes use of A.dll. We use a third-party vendor product (a backup program, but again this is not important) that has installed a Context Menu Handler control under HKEY_CLASSES_ROOT\Directory\shellex\ContextMenuHandlers. Let's say the path to this is c:\Program Files\Vendor\control.dll.

Now, the issue is that when my program opens a file chooser dialog (it's a Qt program that uses QFileDialog which then uses the standard Windows one), this control is loaded to provide context-sensitive right-click functionality. This control depends on a different version of "A.dll" and when control.dll is loaded, my program promptly crashes.

I don't need this extra functionality. What I would love to do is to prevent this specific dll (control.dll) from loading in my process. In an object-oriented world I would simply overload LoadLibrary(), check for this specific DLL, and then call the standard one otherwise. However this doesn't seem feasible.

Is there an easy way to do this?

Thanks! Dan

Answer 1

To prevent the vendor.dll from loading you can use a hook on the following Win32API function LoadLibrary and LoadLibraryEx which are responsible for dynamically loading DLLs and which are also used to load shell extensions. The hook is really assembler code at the code site of the LoadLibrary function, which redirects (jumps) to a function defined by yourself. In this function you can then intercept any call to vendor.dll being loaded and just return 0, which indicates that the library could not be loaded.

Some example code how to go about it using MinHook library:

HMODULE WINAPI LoadLibraryA_check(_In_ LPCTSTR lpFileName)
{
  if (isInWhiteList(lpFileName))
    return loadLibraryA_Original(lpFileName);
  else 
  {
    // Pretend that the module was not found by returning
    // 126 (0x7E): "The specified module could not be found."
    SetLastError(ERROR_MOD_NOT_FOUND);         
    return NULL;
  }
}

bool installWhitelistFilter()
{
  // Initialize MinHook.
  if (MH_Initialize() != MH_OK)
    return false;

  if (MH_CreateHook(&LoadLibraryA, &LoadLibraryA_check, 
      reinterpret_cast<LPVOID*>(&loadLibraryA_Original)) != MH_OK)
    return false;

  if (MH_EnableHook(&LoadLibraryA) != MH_OK)
    return false;

  // same for LoadLibraryW, LoadLibraryExW, LoadLibraryExA

  return true;
}

Answer 2

In this situation, you are launching a standard Windows Open File dialog that internally hosts Windows Explorer as part of its UI, and Explorer loads Shell Extension DLLs. To prevent that, you have to either:

1. call the GetOpenFileName() API directly, opting to use the old-style dialog instead of the newer explorer-style dialog by omitting the OFN_EXPLORER flag (the newer IFileOpenDialog API does not support this option). The older-style dialog does not support Shell Extensions (but it also has an outdated look-and-feel).

2. don't use the standard Open File dialog at all. Make you own instead.

Another option is to let Explorer do its job normally, but create and register your own shim DLL as a replacement for the offending Shell Extension. Have the shim check if the calling process is explorer.exe. If so, load and delegate all actions to the original Extension, otherwise do nothing. See How To Disable Shell Extension In FileOpen Dialog.

Another option might be to create a side-by-side assembly manifest for the offending Shell Extension, and have that manifest explicitly state which DLL to load, so the Extension loads its own version of the DLL and not the version your app is using.