The solution was to use type library marshaling (also known as universal marshaling), using the system-provided marshaling engine in oleaut32.dll. The universal marshaler can be used with any custom interface, provided that the parameters are variant-compatible.
To use the universal marshaler for an interface, the interface is tagged with oleautomation, as in this sample interface:
[
object,
oleautomation,
uuid(23D4EC0B-96DA-4D18-82BD-40E3AA0483FD),
version(1.0),
dual,
helpstring("Description of ICustomInterface1"),
pointer_default(unique)
]
interface ICustomInterface1 : IDispatch
{
// Methods and properties …
}
Universal marshaling also requires the following Registry entries:
HKCR\TypeLib<type library GUID><type library version>
HKCR\TypeLib<type library GUID><type library version>\0
HKCR\TypeLib<type library GUID><type library version>\0\win32 = “path to type library file”
HKCR\TypeLib<type library GUID><type library version>\Flags
HKCR\TypeLib<type library GUID><type library version>\HelpDir = “path to help folder”
[HKEY_CLASSES_ROOT\Interface<interface GUID> @="Interface name"
[HKEY_CLASSES_ROOT\Interface<interface GUID>\ProxyStubClsid32
="{00020424-0000-0000-C000-000000000046}"
HKCR\Interface<interface GUID\TypeLib = type library GUID
With this type information, the universal marshaler creates a proxy-stub at runtime, eliminating the need for a custom proxy-stub DLL.
In our case, given that the components were ATL-based, the Registry entries were created automatically once the interfaces were tagged with ‘oleautomation’ and the components rebuilt. For non-ATL projects, this can be done programmatically with the Win32 API’s LoadTypeEx() (which I haven't tried).
Once this was done, non-default interfaces of our ATL-based components became available out-of-process to our migrated-to-VB.NET Excel COM add-in.
All this is detailed in Chapter 5 of "Developer's Workshop to COM and ATL 3.0."