2

I'm in the process of implementing a COM server in an EXE file. To be precise, I'm adding a COM interface to an existing application. with the ultimate goal of automating the application.

The first component and an interface (with a single method so far) are basically in place. I can build an in-proc server in a DLL and successfully obtain an interface pointer and call the method. This was done as a first step since I'm only just learning. I won't need the DLL later; it just serves as proof of concept that my component and interface are basically ok.

Then I built an out-of-process server in an EXE. I have got to the point where I can call CoCreateInstance() from the client, and the EXE is started, registers its factory, and the factory creates an instance of the component. CoCreateInstance returns S_OK and the client receives an interface pointer that is not NULL.

The problem occurs when I call the interface's method.

  • First, the breakpoint in the method is not hit (yes, it's in another process, but I'm also debugging the server. Other breakpoints in the server are hit ok).
  • Second, the client reports Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention..

I'm absolutely aware that methods in a COM interface must use __stdcall, and I have checked several times that it's not missing. Besides, the implementation of the component (C++) derives from the interface generated by MIDL. So the interface's header file has the correct calling convention, and if the component's header file did not, the compiler would complain about the override differing.

Since the breakpoint is not even hit, my gut feeling is that something is fundamentally wrong with the local procedure call, along the lines of the vtbl not being set up correctly. Does anyone have a suggestion what could be causing the observed behaviour? Any tips on how to debug the proxy/stub code?

EDIT:

In response to WhozCraig, here the IDL file:

import "unknwn.idl";

// Interface IMyApp1
[
   object,
   uuid(440EA043-DF6D-4df9-963D-7660BBA829EF),
   helpstring("IMyApp1 Interface"),
   pointer_default(unique)
]

interface IMyApp1: IUnknown
{
   HRESULT ShowAboutBox(void);
}
Cody Gray - on strike
  • 239,200
  • 50
  • 490
  • 574
Dabbler
  • 9,733
  • 5
  • 41
  • 64
  • This sounds like the proxy/stub for marshaling the interface are not wired up correctly. Are you building a proxy-stub? Seeing the IDL declaration of the interface including its attributes may be nice as well, if you can post it. – WhozCraig Oct 29 '12 at 20:25
  • Yes, I'm building a proxy/stub DLL. MIDL generates dlldata.c, xyz_h.h, xyz_p.c, and xyz_i.c, from which I build the DLL. The component's implementation in C++ derives from xyz.h. – Dabbler Oct 29 '12 at 21:02
  • Ok. last dumb question (and forgive me, but you'd be amazed sometimes), are you *registering* the proxy/stub? – WhozCraig Oct 29 '12 at 21:04
  • Absolutely valid question, but yes, it's registered. CoCreateInstance would return an error otherwise, but it succeeds, and when debugging the server, I can see that all the Factory::CreateInstance stuff runs fine. – Dabbler Oct 29 '12 at 21:06
  • It sounds like you have everything setup correctly. Next dumb question, does it *start* your DCOM server upon CoCreateInstance in your client. Running the server manually should work so long as it sets up the proper class factory as running and registers it with COM, but just to be sure it isn't trying to start a *different* instance, have you tried *not* running your server and seeing if the CCI call to create the object properly *starts* it. (forgive the grasping, but DCOM-launch-debugging is always slow and painful). – WhozCraig Oct 29 '12 at 21:14
  • Never mind last comment. I can see your post says this is correctly happening.. still pondering... and btw, your IDL looks fine. The library section follows, I assume, and sets up the coclass correctly with correct default interface spec. – WhozCraig Oct 29 '12 at 21:16
  • OK. my list of lame questions continues. You say you tested this in a DLL prior to your moving it to an EXE. On the off-chance you used the SAME IDL file and moved it to your new project, prior to building and registering your LocalServer32 DCOM server, did you blow away the InProcServer32 entry for the same CLSID ? Of course issuing a whole new set of CLSIDs and IID's makes this moot, but if you did, in fact, use the same IDL file, I *think* I know what could cause your current behavior. – WhozCraig Oct 29 '12 at 21:29
  • @WhozCraig: The IDL file as shown is all I have, it doesn't contain a library section. Since it's the first IDL file I wrote, it may be missing something. But I think a library section is only required when building a TLB, which I don't do yet (I prefer to get a basic EXE server running first). The DLL server doesn't use an IDL file. For the DLL server I originally hand-coded the interface. Now, instead, it uses the same abc_h.h file as the EXE server. This was to see whether using abc_h.h was part of the problem, but the DLL server still works. Btw, the EXE server is local - no DCOM. – Dabbler Oct 29 '12 at 21:45
  • 1
    Both mishaps have a simple explanation. The client is calling *another* function in the interface. Which will happen when the client and the server are built with different IDL files. Or more typically: when one of them was built with an old version of the IDL. – Hans Passant Oct 29 '12 at 22:23
  • 1
    @Dabbler fyi, an EXE server *is* DCOM, just not all the way out to another machine. Hans is entirely correct that using a mismatched IDL from the client will cause severe issues like this, but I'm really curious to know if you cleaned out the InProcServer32 entry for the CLSID you're registering with your EXE server (and were registering with your DLL server) and if not, at least make sure the client does NOT have CLSCTX_INPROC (which is part of CLSCTX_ALL) as the creation-disposition for the object being manufactured. I just want to ensure your DLL isn't begin loaded by mistake. – WhozCraig Oct 29 '12 at 22:45
  • 1
    I've setup a link to a sample that you can compare against your existing project. It is about as simple as you can get, a [DCOM Server/Proxy/Client](http://www.filedropper.com/dcomtest) solution for VS2010. Hopefull it will get you going. Wish I could help more. Note: the GREY box, *not* the green one for the download (hate sites that do that). – WhozCraig Oct 29 '12 at 23:32
  • Thanks for your help so far, I really appreciate it! I have completely eliminated the DLL (deregistered it, cleaned it, removed its project from the solution, renamed the source file folder, checked the registry for any traces, booted the PC, verified it's not loaded with Process Explorer). Unfortunately, the problem persists. I currently don't have a clue what I'm doing wrong; any help is most welcome. I'm using VS2005, so couldn't open the solution files you kindly provided, but I'll examine the sources shortly. – Dabbler Oct 30 '12 at 20:38

1 Answers1

1

I found the problem. It's quite an embarrassing mistake, but interesting to know that it leads to the observed effect, so I'll post it here in case someone else has the same problem.

The client was doing

HdResult = CoCreateInstance(
   sClassIdApp,
   NULL,
   CLSCTX_LOCAL_SERVER,
   IID_IUnknown,   // Oops...
   (void**) &pInterface);

pInterface->ShowAboutBox();

instead of

HdResult = CoCreateInstance(
   sClassIdApp,
   NULL,
   CLSCTX_LOCAL_SERVER,
   IID_IMyApp1,
   (void**) &pInterface);

pInterface->ShowAboutBox();

Duh...

Dabbler
  • 9,733
  • 5
  • 41
  • 64