How to absolutely ensure Direct3D9 hook overlay rendering on top

Question

I'm trying to hook IDirect3DDevice9::Present or ::EndScene and render my own overlay (which, for now, is just a simple rectangle) on top of everything else in a D3D9 application, but my overlay seems to be appearing and disappearing quite randomly. The drawing code I'm currently using is:

typedef struct CUSTOMVERTEX {
    float x, y, z, rwh;
    DWORD color;
};

#define CUSTOMFVF (D3DFVF_XYZRHW | D3DFVF_DIFFUSE)

void draw() {
    // the positions are just for testing purposes, so they don't really make sense
    CUSTOMVERTEX vertices[] = {
        { 0, 0, 0, 1.0f, D3DCOLOR_XRGB(255, 255, 255) },
        { 0, cursor_pos.y+500, 0, 1.0f, D3DCOLOR_XRGB(127, 255, 255) },
        { cursor_pos.x, cursor_pos.y, 0, 1.0f, D3DCOLOR_XRGB(255,255, 255) },
        { cursor_pos.x, 600, 0, 1.0f, D3DCOLOR_XRGB(127, 0, 0) }
    };

    if (vBuffer == 0) return;

    VOID* pVoid; 

    vBuffer->Lock(0, 0, &pVoid, 0); 
    memcpy(pVoid, vertices, sizeof(vertices));    
    vBuffer->Unlock(); 

    d3dDevice->SetRenderState(D3DRS_LIGHTING, FALSE);
    d3dDevice->SetRenderState(D3DRS_ALPHABLENDENABLE, FALSE);

    D3DMATRIX orthographicMatrix;
    D3DMATRIX identityMatrix;

    // MAKE_D3DMATRIX should be equivalent to D3DXMATRIX constructor
    D3DMATRIX viewMatrix = MAKE_D3DMATRIX( 
        1, 0, 0, 0,
        0, 1, 0, 0,
        0, 0, 1, 0,
        ((float)-(get_window_width()/2)), ((float)-(get_window_height()/ 2)), 0, 1
    );
    // MAKE_ORTHO_LH is equivalent to D3DMatrixOrthoLH
    MAKE_ORTHO_LH(&orthographicMatrix, (FLOAT)get_window_width(), (FLOAT)get_window_height(), -1.0, 1.0); 

    // and this to D3DMatrixIdentity
    MAKE_IDENTITY(&identityMatrix); // and this to D3DMatrixIdentity

    d3dDevice->SetTransform(D3DTS_PROJECTION, &orthographicMatrix);
    d3dDevice->SetTransform(D3DTS_WORLD, &identityMatrix);
    d3dDevice->SetTransform(D3DTS_VIEW, &viewMatrix);

    d3dDevice->SetRenderState(D3DRS_ZENABLE, false);
    d3dDevice->SetFVF(CUSTOMFVF);
    d3dDevice->SetStreamSource(0, vBuffer, 0, sizeof(CUSTOMVERTEX));
    d3dDevice->DrawPrimitive(D3DPT_TRIANGLEFAN, 0, 2);
    d3dDevice->SetRenderState(D3DRS_ZENABLE, true);
}

I can't seem to figure out why this would cause the overlay to pop in and out of existence at seemingly random points in time.. What am I missing?

Progress: calling `SetVertexShader(NULL)` and `SetPixelShader(NULL)` fixes the problem in SOME circumstances, but not all :( — ehoopz, Jun 02 '17 at 10:18

score 0 · Accepted Answer · answered Jun 04 '17 at 11:24

I found a foolproof method to render my stuff directly to the backbuffer (on CPU):

IDirect3DSwapChain9 *sc;
if (FAILED(d3dDevice->GetSwapChain(0, &sc))) {
    PRINT("GetSwapChain failed\n");
    return;
}

IDirect3DSurface9 *s;
if (FAILED(sc->GetBackBuffer(0, D3DBACKBUFFER_TYPE_MONO, &s))) {
    PRINT("GetBackBuffer failed\n");
    return;
}

D3DLOCKED_RECT r;
if (FAILED(s->LockRect(&r, NULL, D3DLOCK_DONOTWAIT))) {
    PRINT("LockRect failed\n");
    return;
}

// here, find out pixel format and manipulate 
// pixel buffer through r->pBits, using r->Pitch accordingly

s->UnlockRect();

s->Release();
sc->Release();

Will most probably incur a performance penalty, but in my application this doesn't really make a difference. A more optimal way would be to harness the GPU's hwacceled rendering capabilities, but I currently lack the D3D knowledge to do so.

How to absolutely ensure Direct3D9 hook overlay rendering on top

1 Answers1