Is Desktop/screenshot capture with screengrab from DirectX Tool Kit possible?

Question

So, getting to the metal right away. I'm writing a c++ program which purpose is to take the screen as an input image and process it with an object detector (from dlib) but I'm having troubles understanding how one can get a pointer to a swapChain and context/content of the screen.

The following code from ScreenGrab (DirectX Tool Kit) seems to be able to do the trick. The issue is that I don't know how to get it to work. I'm using NuGet in visual studio 2017 to get the library (called directxtk_uwp, there is also one called directxtk_desktop_2015 which I am not using.) and I'm getting the following six errors:

• error C2065: 'swapChain': undeclared identifier
• error C2227: left of '->GetBuffer' must point to class/struct/union/generic type
• error C2065: 'immContext': undeclared identifier
• error C2228: left of '.Get' must have class/struct/union
• error C2653: 'DX': is not a class or namespace name
• error C3861: 'ThrowIfFailed': identifier not found

from running the inserted example code from the ScreenGrab wiki-page:

#include <ScreenGrab.h>
#include <wrl\client.h>
#include <Wincodec.h>


int main() {


    using namespace DirectX;
    using namespace Microsoft::WRL;

    ComPtr<ID3D11Texture2D> backBuffer;
    HRESULT hr = swapChain->GetBuffer(0, __uuidof(ID3D11Texture2D),
        reinterpret_cast<LPVOID*>(backBuffer.GetAddressOf()));
    if (SUCCEEDED(hr))
    {
        hr = SaveWICTextureToFile(immContext.Get(), backBuffer.Get(),
            GUID_ContainerFormatJpeg, L"SCREENSHOT.JPG");
    }
    DX::ThrowIfFailed(hr);


        return 0;
}

I have really just started out programming with c++ and came from Java programming, which I've been actively developing in for a year now. So keeping it simple to understand would be appreciated ^^

Just to be clear, I would like something rather fast. I have gotten a screengrab to work with GDI with the following code:

#include <iostream>
#include <Windows.h>
#include <Wincodec.h>
#include <ctime>
#include <cstring>
#include <atlimage.h>
#include <d3d9.h>
using namespace std;


int main()
{
    try
    {
        int nScreenWidth = GetSystemMetrics(SM_CXSCREEN);
        int nScreenHeight = GetSystemMetrics(SM_CYSCREEN);
        HWND hDesktopWnd = GetDesktopWindow();
        HDC hDesktopDC = GetDC(hDesktopWnd);
        HDC hCaptureDC = CreateCompatibleDC(hDesktopDC);

        for (int i = 0; i < 10;i++) {
            clock_t begin = clock();
            HBITMAP hCaptureBitmap = CreateCompatibleBitmap(hDesktopDC,
                nScreenWidth, nScreenHeight);
            SelectObject(hCaptureDC, hCaptureBitmap);
            BitBlt(hCaptureDC, 0, 0, nScreenWidth, nScreenHeight,
                hDesktopDC, 0, 0, SRCCOPY | CAPTUREBLT);
            CImage image;
            image.Attach(hCaptureBitmap);
            image.Save("test.jpg");
            DeleteObject(hCaptureBitmap);
            cout << double(clock() - begin) / (clock_t)1000 << endl;
        }
        ReleaseDC(hDesktopWnd, hDesktopDC);
        DeleteDC(hCaptureDC);
        IDirect3DSurface9 *surface;
        system("pause");
    }
    catch (exception& e)
    {
        cout << "\nexception thrown!" << endl;
        cout << e.what() << endl;
        system("pause");
    }
}

which results in the following:

0.091
0.05
0.052
0.06
0.047
0.05
0.057
0.06
0.06
0.051

and as you can see, that really only gets my up to 1/0.47 ≈ 21 frames per second. I have also seen this answer, but yet again, I have no clue how to get the context and swapChain of the screen. Thank you.

Answer 1

So after a little more looking and with the guidance of the great Chuck! I've found a solution that is about 3.5 times as fast at saving screenshots to disk (if you remove drawing mouse and the overhead of having 8 milliseconds in the beginning of the grabImage function). You can see the code here. The following is the resulting time in milliseconds of saving ten images to the disk:

0.015
0.017
0.012
0.021
0.012
0.019
0.012
0.022
0.018
0.015

Summing these up and the others from the GDI program, we get 0.578/0.163≈3.546 times the speed. Keep in mind that I have replaced this part:

// Copy image into GDI drawing texture
        lImmediateContext->CopyResource(lGDIImage, lAcquiredDesktopImage);

        // Draw cursor image into GDI drawing texture
        CComPtrCustom<IDXGISurface1> lIDXGISurface1;

        hr = lGDIImage->QueryInterface(IID_PPV_ARGS(&lIDXGISurface1));

        if (FAILED(hr))
            return false;

        CURSORINFO lCursorInfo = { 0 };

        lCursorInfo.cbSize = sizeof(lCursorInfo);

        auto lBoolres = GetCursorInfo(&lCursorInfo);

        if (lBoolres == TRUE)
        {
            if (lCursorInfo.flags == CURSOR_SHOWING)
            {
                auto lCursorPosition = lCursorInfo.ptScreenPos;

                auto lCursorSize = lCursorInfo.cbSize;

                HDC  lHDC;

                lIDXGISurface1->GetDC(FALSE, &lHDC);

                DrawIconEx(
                    lHDC,
                    lCursorPosition.x,
                    lCursorPosition.y,
                    lCursorInfo.hCursor,
                    0,
                    0,
                    0,
                    0,
                    DI_NORMAL | DI_DEFAULTSIZE);

                lIDXGISurface1->ReleaseDC(nullptr);
            }

        }

        // Copy image into CPU access texture
        lImmediateContext->CopyResource(lDestImage, lGDIImage);

with the following line:

lImmediateContext->CopyResource(lDestImage, lAcquiredDesktopImage);

since I did not need the cursor. In fact, it might be worse to have it considering I'll be detecting things on the screen. As mentioned before, I have also relocated the line Sleep(8); // not sure if required from here:

    hr = lDevice->CreateTexture2D(&desc, NULL, &lDestImage);

    if (FAILED(hr))
        return false;

    if (lDestImage == nullptr)
        return false;

    return true;
}

bool grabImage() {
    int lTryCount = 4;

    do
    {
        Sleep(8); // not sure if required

to here, in the init function:

    hr = lDevice->CreateTexture2D(&desc, NULL, &lDestImage);

    if (FAILED(hr))
        return false;

    if (lDestImage == nullptr)
        return false;
    Sleep(8);
    return true;
}

bool grabImage() {
    int lTryCount = 4;

    do
    {
            //Sleep(5); // not sure if required

You could remove it completely as well, although that seems to cause "blank"/black image files in the beginning. For me, there were only two blank images before it began to capture. With the delay after the whole init function, I'm getting the first images as well without the overhead. Thank you.