Figuring out the memory layout of objects without debugging a running program?

Question

I'm working on reverse engineering a program. I'm using IDA Pro and the Hex-Rays Decompiler. I'm coming up against chunks of code where I know there is an object and there is a method call on the object but it's shown in a way I'm not able to figure out. For example:

 if ( (*(*interfacePtr + 24))(interfacePtr, &v23) >= 0 )

I know here that interfacePtr is pointing to a ICLRRuntimeHost object. (C++, .NET CLR runtime) However.... I have no idea what is at *(*interfacePtr + 24). I can tell it's a method but how do I figure out what is sitting at +24?

Don't you think this s better answered on one of the reverse engineering forums? Or perhaps the IDA/hex-rays forum, which you should have access to after paying the hefty pricetag for this software? — snemarch, Aug 12 '12 at 03:03
IDA can parse C header files for structure declarations. `ICLRRuntimeHost` is defined in the Windows SDK `mscoree.h` file, with some work you should be able to either feed this file to IDA or at least extract the class and vtable declarations to another file and parse that. After IDA has a definition of this class, you can simply tell Hex-Rays that `interfacePtr` is of the type `ICLRRuntimeHost` (press Y or right-click the variable name) and it will update the display to include the function name. — DCoder, Aug 12 '12 at 05:38
@snemarch I think this question is perfectly valid considering SO is all about programming, which, at the end of the day.... is what my question is really about. I would use the discussion boards for a product if I had an issue with a product. However, this is simply me not having enough knowledge about the underlying mechanics of a native Windows executable... not me having an issue with IDA or the Hex-Rays decompiler. — Toby Lawrence, Aug 12 '12 at 16:07
@DCoder I have been simply correlating the vtable to the method offset specified in the calls for the moment... but I will definitely give your approach a shot. An automatic association would be much much nicer. :) — Toby Lawrence, Aug 12 '12 at 16:09

Michael · Accepted Answer · 2012-08-13T05:38:32.807

The vtable of a class is just a list of pointers to functions. It contains one pointer for each virtual function, in the order: very, very top base class, next base class, subclass of that, . . . most derived class.

Example:

struct A {
    virtual ~A() {}
    virtual void foo() = 0;
}
struct B : public A {
    virtual void foo() { // do something }
    virtual void bar() { // do something else }
}

The vtable for B would contain, in this order:

~A
foo
bar

(The ones for A have to come first, so that the same vtable can be used by parts of the code that have a pointer of type A to this object; that code doesn't know that the underlying object is really B.)

If you're looking at 32-bit source, pointers are 4 bytes, so 24 = 4 * 6, you're looking at the 7th virtual function (index starts at 0). If you're on 64-bit, pointers are 8 bytes, so 24 = 8 * 3, and you're looking for the 4th. Actually, I haven't used the IDA "convert to C++" functionality, so maybe 24 is actually the 24th entry in the table.

Easy way to confirm: write your own program. Declare a variable of type ICLRRuntimeHost. Call the function you suspect (based on looking at the header file and counting to 7 or 4, depending on bitness, or 24, if I misunderstood your example). Look at the generated assembly code, and confirm whether your index is right. (I'm always off by one on that kind of thing, so this will provide a check.)

Am I right in assuming that you meant to declare B extending A? — Toby Lawrence, Aug 12 '12 at 14:36
Aside from my above comment, looking at mscoree.h while reading this psuedocode is matching up nicely as far as the whole vtable ordering scheme. Looks like that was the big missing link for me. :D — Toby Lawrence, Aug 12 '12 at 15:24

score 0 · Answer 2 · answered Sep 12 '12 at 08:14

Have a look at the definitions of ICLRRuntimeHostVtbl and ICLRRuntimeHostVtbl in mscoree.h

roughly translated to something hexrays would understand, they look like this:

struct ICLRRuntimeHost {
    ICLRRuntimeHostVtbl *vtbl;
};
struct ICLRRuntimeHostVtbl {
    _DWORD (*QueryInterface)(ICLRRuntimeHost*, _DWORD*, void**);
    _DWORD (*AddRef)(ICLRRuntimeHost*);
    _DWORD (*Release)(ICLRRuntimeHost*);
    _DWORD (*Start)(ICLRRuntimeHost*);
    _DWORD (*Stop)(ICLRRuntimeHost*);
    _DWORD (*SetHostControl)(ICLRRuntimeHost*, void*);
    _DWORD (*GetCLRControl)(ICLRRuntimeHost*, void**);
};

Your variable interfacePtr should have type: ICLRRuntimeHost, then your code should decompile like this:

interfacePtr->GetCLRControl(&v23);

Figuring out the memory layout of objects without debugging a running program?

2 Answers2