DirectX hook ignores vertex colors

Question

I have detoured a detour from a game's third-party overlay to draw on top of it and I managed to do as much, just without color. I had no prior experience with DirectX so I'm having a really bad time to figure out how to do anything at all.

By following "http://www.directxtutorial.com/"'s DirectX 9 tutorials I managed to draw a triangle, but mine has no color no matter what color I set the vertex colors to be!

This is my hook and attempt to draw a colored triangle (which ends up white):

    struct DXVertex
    {
        FLOAT x, y, z, rhw;
        D3DCOLOR color;
    };
    
    LPDIRECT3DVERTEXBUFFER9 v_buffer{ nullptr };
    
    void initMyStuff(IDirect3DDevice9* dxDevice)
    {
        DXVertex vertexArray[] =
        {
            {350.0f, 50.0f, 0.5f, 0.0f, D3DCOLOR_XRGB(255, 0, 0)}, // All colors end up being white
            {520.0f, 400.0f, 0.5f, 0.0f, D3DCOLOR_XRGB(0, 255, 0)},
            {120.0f, 400.0f, 0.5f, 0.0f, D3DCOLOR_XRGB(0, 0, 255)}, // This doesn't make a difference either
        };
    
        dxDevice->CreateVertexBuffer(3 * sizeof(DXVertex), NULL, D3DFVF_XYZRHW | D3DFVF_DIFFUSE, D3DPOOL_DEFAULT, &v_buffer, NULL);
        void* bData;
    
        v_buffer->Lock(NULL, NULL, &bData, NULL);
        memcpy(bData, vertexArray, sizeof(vertexArray));
        v_buffer->Unlock();
    }
    
    void drawTriangle(IDirect3DDevice9* dxDevice, float x0, float y0, float width, float height, D3DCOLOR color)
    {   
        dxDevice->SetTexture(0, NULL);
        dxDevice->SetPixelShader(NULL); // If this isn't here, the triangle doesn't draw
        // dxDevice->SetVertexShader(NULL); // If this is here, the triangle doesn't draw either
    
        // Settings I tried to mess with to no avail, I've tried all independently as well
        dxDevice->SetRenderState(D3DRS_COLORVERTEX, TRUE);
        dxDevice->SetRenderState(D3DRS_LIGHTING, FALSE);
        dxDevice->SetRenderState(D3DRS_ALPHABLENDENABLE, TRUE);
        dxDevice->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);
        dxDevice->SetRenderState(D3DRS_SRCBLENDALPHA, D3DRS_DESTBLENDALPHA);
        
        dxDevice->SetFVF(D3DFVF_XYZRHW | D3DFVF_DIFFUSE);
        dxDevice->SetStreamSource(0, v_buffer, 0, sizeof(DXVertex));
        dxDevice->DrawPrimitive(D3DPT_TRIANGLELIST, 0, 1);
    
    }
    
    // The overlay hook
    HRESULT STDMETHODCALLTYPE present(IDirect3DDevice9* thisptr, const RECT* src, const RECT* dest, HWND wnd_override, const RGNDATA* dirty_region) 
    {
        if (!v_buffer)
            initMyStuff(thisptr);
        
        thisptr->BeginScene();
    
        // I am not even the parameters as of now for testing.
        drawTriangle(thisptr, 0.0f, 0.0f, 0.0f, 0.0f, 0);
    
        thisptr->EndScene();
    
        // Call original
        return original_present(thisptr, src, dest, wnd_override, dirty_region);
    }

Something I noticed in the disassembler is that the third-party overlay hooks "Direct3DDevice9::SetGammaRamp", so I suspected that could be my problem so I tried printing values from "Direct3DDevice9::GetGammaRamp", which apparently isn't hooked.

Below is the picture of the output triangle that should have color and the gamma ramp of each color from the swap chain 0:

I have been at this for a huge while now, and I hope I won't need to end up buying an ancient DirectX 9 book to get a grasp on the situation...

UPDATE AFTER SOME REVERSING: These are the functions from the directX device that the overlay hooks and what they're interested in:

• IDirect3DDevice9::SetSamplerState when D3DRS_SRGBWRITEENABLE is to be changed.
• IDirect3DDevice9::SetRenderState when D3DRS_SRGBWRITEENABLE is to be changed.
• IDirect3DDevice9::SetGammaRamp

All of these are related to gamma modification, I assume it has to do with dimming the background (which it does) when the overlay is shown. It appears to let all the parameters pass through to the device without modification though.

Answer 1

Apologies ahead of time for "Answering" and not commenting, but I'm a new contributor. I did rigorously devour Frank Luna's introduction to Direct X 9 back in the day which had all this gritty low level stuff and built a render engine from scratch. That said, I can't pinpoint your problem off the top of my head and go "That's it! I recognize it immediately!"

Factoring in all that, my baseline assumption and best guess is that your third party overlay initialized DirectX and set a bunch of PresentParameters and RenderStates conducive to rendering with lighting, normals, or textures. Something, somewhere, earlier in this code is clearly getting set before you ever touch it, and you're trying to unset it. If you can find where that is, you can study what it did and attempt to reverse it.

Without seeing the rest of the third party overlay and how it is interacting with your code, ie: what is calling what, it's hard to judge where the culprit is.

For example, I can't see Main.cpp, or wherever DirectX is being initialized. You aren't following the organizational and function naming strategies of the tutorial, which is leading to confusion about what parts of the overlay are calling your program when.

There are also some organizational concerns I have. You repeatedly recreate and then release a vertex buffer right before drawing it, instead of separating the code into an initialization and a render function like your tutorials do; and then you use the word 'present' as a function name where the tutorials use renderScene() and in which the tutorials call the actual D3D present function, which made me a little dizzy.

Let us know if you make any future discoveries or more clues, someone will help you. Also you might want to tag this with gary's mod, rather than just call it a third party overlay, as it's possible that'll net you some help.

Answer 2

I got it! I tried using Dear ImGui (https://github.com/ocornut/imgui) and it worked fine, so I dug into the dxd9 initialization it did and copied all the setup settings:

dxDevice->SetPixelShader(NULL);
dxDevice->SetVertexShader(NULL);
dxDevice->SetRenderState(D3DRS_CULLMODE, D3DCULL_NONE);
dxDevice->SetRenderState(D3DRS_LIGHTING, FALSE);
dxDevice->SetRenderState(D3DRS_ZENABLE, FALSE);
dxDevice->SetRenderState(D3DRS_ALPHABLENDENABLE, TRUE);
dxDevice->SetRenderState(D3DRS_ALPHATESTENABLE, FALSE);
dxDevice->SetRenderState(D3DRS_BLENDOP, D3DBLENDOP_ADD);
dxDevice->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_SRCALPHA);
dxDevice->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);
dxDevice->SetRenderState(D3DRS_SCISSORTESTENABLE, TRUE);
dxDevice->SetRenderState(D3DRS_SHADEMODE, D3DSHADE_GOURAUD);
dxDevice->SetRenderState(D3DRS_FOGENABLE, FALSE);
dxDevice->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_MODULATE);
dxDevice->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
dxDevice->SetTextureStageState(0, D3DTSS_COLORARG2, D3DTA_DIFFUSE);
dxDevice->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
dxDevice->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
dxDevice->SetTextureStageState(0, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);
dxDevice->SetSamplerState(0, D3DSAMP_MINFILTER, D3DTEXF_LINEAR);
dxDevice->SetSamplerState(0, D3DSAMP_MAGFILTER, D3DTEXF_LINEAR);

and there it was:

Hope this can be of use for someone in the future.