EndScene hook questions

Question

So recently I wanted to add an imgui interface to an example window using DirectX, so I watched on a video I had to hook the EndScene function using DirectX9sdk to be able to add my custom imgui interface.

However I have some questions:

1. Where can I find any documentation for the DirectX9 functions and types,( if there is any, because I do not understand why we specifically have to hook the EndScene function) or where could I find any article explaining more in depth how directX works?
2. I've seen two version so far of EndScene hooks one with a patternScanning function which scans a signature in the shaderapi dll and another which creates a DirectXDevice and then accesses the vtable from there; are there any sources online, or is it something we have to do ourselves? Here is the version I have:

while (!DirectXDevice) // loops until it finds the device
        DirectXDevice = **(DWORD**)(FindPattern("shaderapidx9.dll", "A1 ?? ?? ?? ?? 50 8B 08 FF 51 0C") + 0x1);

    void** pVTable = *reinterpret_cast<void***>(DirectXDevice); // getting the vtable array
    oEndScene = (f_EndScene)DetourFunction((PBYTE)pVTable[42], (PBYTE)Hooked_EndScene)//getting the 42th virtual function and detouring it to our own

4. I don't really understand what __stdcall does here, I do know it is used to call WINAPI functions but what for here?

HRESULT __stdcall Hooked_EndScene(IDirect3DDevice9* pDevice){//some code}

Note: thats the function I hook to the original endscene.

Thank you really much, I'm sorry if there are alot of questions but I really can't wrap my head around this.

Answer 1

How do you know which functions you need to hook?

To put it bluntly, you have to be an experienced DirectX graphics programmer to find that out. Don't expect being able to hook into a framework that you don't understand. It just so happens that EndScene will always be called after all the other draw calls on the render target.

There are tons of D3D9 programming resources available, online and in paper form. Most of them are not free. I'm afraid this is not the answer you were hoping for.

What is the deal with pattern scanning, or creating a temporary D3D9 device?

Microsoft did not put any explicit effort into making EndScene hookable. It just happens to be hookable because every normal function is hookable. You need a way to find the function in memory, because the function will not always be at the same address.

One approach is to scan for known instructions that appear inside the function. Someone needs to be the first person to find out that pattern that you can scan for. You are far from the first person to hook EndScene, so many have reverse-engineered the function before and shared searchable patterns.

NOTE: The pattern does not necessarily need to be directly inside the target function. It might also lead you to something else first, in your case, the ID3D9Device instance. The important thing is that you can find your way to the EndScene function somehow.

Another approach is to get a pointer to the function. If it was a regular C function, that would be easy. It's hard here because OOP tends to make these things hard - you have to fight your way through various interfaces to get the correct vtable.

Both methods have advantages and disadvantages -- creating a D3D9 device is safer, but also more intrusive, because the target process might not expect someone to just randomly create new devices.

Why does the hook function need __stdcall?

Since you replace the original function with your hooked version, the calling convention of the hooked function must be the same as the calling convention of the original function. The caller of EndScene expects (and was compiled with) a __stdcall convention, so the new function must also behave the same way, otherwise the stack will be corrupted. Your act of replacing the function does not change the way the caller calls it.