0

I have been struggling with this issue for the entire day and I am still not making any progress.

CONTEXT

I am trying to reproduce a virtual piano keyboard (driven with MIDI files) on LEDs attached to a physical keyboard so that my wife can learn how to play piano quicker (Christmas gift). Here is a picture of the program: enter image description here

WHAT THE CODE DOES

(Or is supposed to do) I enter the coordinates of the keyboard as measured in Paint.NET after taking a screenshot of the program full-screen. The user is asked to put the software full screen. Then I calculate the coordinates of the middle of all the keys, and get the colour of the key when it is not activated. Finally I check that all white keys have the same colour, and that the black keys have the same colour (to make sure the keyboard is correctly displayed). Later, I'll loop forever, getting the colour of each key and lighting up the corresponding LED on the physical keyboard if the colour of that key has changed from the "rest" state.

PROBLEM

I know I pick the colours at the right location, because SetPixel() displays the red pixels spot on in the middle of all keys (as seen in code). However, GetPixel still returns wrong values often times (1/4th of the time for the white keys, always for the black keys). What's happening ?

WHAT I TRIED

A added a manifest file to the project which successfully modified the size of the desktop to the correct value of 1920x1080 - but even though the RGB values changed a bit, they are still off - I added a captureAndSave() function in case the computer "sees" a different thing than me when I do a screen capture (to obtain all the input coordinates), or than whatever does SetPixel() because this works - but the image is indeed identical, so I don't know what is happening. - Many debugging runs...

CODE

(skip to VirtualKeyboard::init())

#include <iostream>
#include <windows.h>

using namespace std;

const unsigned int N_WHITE_KEYS = 49;
const unsigned int N_PIANO_KEYS = N_WHITE_KEYS+35;
const unsigned int N_ABSENT_BLK_KEYS = N_WHITE_KEYS*2-N_PIANO_KEYS-1;
const unsigned int ABSENT_BLK_KEYS[N_ABSENT_BLK_KEYS] = {3,7,10,14,17,21,24,28,31,35,38,42,45}; //TODO: find a formula rather than this list

const unsigned int VKxLeft = 19,
                   VKyTop = 150,
                   VKxRight = 1731,
                   VKyBtm = 324;


//TEMP
#include <stdio.h>

bool captureAndSave(const HWND& hWnd, int nBitCount, const char* szFilePath)
{
    if(!szFilePath || !strlen(szFilePath))
    {
        cout << "bad function arguments\n";
        return false;
    }

    //calculate the number of color indexes in the color table
    int nColorTableEntries = -1;
    switch(nBitCount)
    {
        case 1:
            nColorTableEntries = 2;
            break;
        case 4:
            nColorTableEntries = 16;
            break;
        case 8:
            nColorTableEntries = 256;
            break;
        case 16:
        case 24:
        case 32:
            nColorTableEntries = 0;
            break;
        default:
            nColorTableEntries = -1;
            break;
    }

    if(nColorTableEntries == -1)
    {
        cout << "bad bits-per-pixel argument\n";
        return false;
    }

    HDC hDC = GetDC(hWnd);
    HDC hMemDC = CreateCompatibleDC(hDC);

    int nWidth = 0;
    int nHeight = 0;

    if(hWnd != HWND_DESKTOP)
    {
        RECT rect;
        GetClientRect(hWnd, &rect);
        nWidth = rect.right - rect.left;
        nHeight = rect.bottom - rect.top;
    }
    else
    {
        nWidth = ::GetSystemMetrics(SM_CXSCREEN);
        nHeight = ::GetSystemMetrics(SM_CYSCREEN);
    }


    HBITMAP hBMP = CreateCompatibleBitmap(hDC, nWidth, nHeight);
    SelectObject(hMemDC, hBMP);
    BitBlt(hMemDC, 0, 0, nWidth, nHeight, hDC, 0, 0, SRCCOPY);

    int nStructLength = sizeof(BITMAPINFOHEADER) + sizeof(RGBQUAD) * nColorTableEntries;
    LPBITMAPINFOHEADER lpBitmapInfoHeader = (LPBITMAPINFOHEADER)new char[nStructLength];
    ::ZeroMemory(lpBitmapInfoHeader, nStructLength);

    lpBitmapInfoHeader->biSize = sizeof(BITMAPINFOHEADER);
    lpBitmapInfoHeader->biWidth = nWidth;
    lpBitmapInfoHeader->biHeight = nHeight;
    lpBitmapInfoHeader->biPlanes = 1;
    lpBitmapInfoHeader->biBitCount = nBitCount;
    lpBitmapInfoHeader->biCompression = BI_RGB;
    lpBitmapInfoHeader->biXPelsPerMeter = 0;
    lpBitmapInfoHeader->biYPelsPerMeter = 0;
    lpBitmapInfoHeader->biClrUsed = nColorTableEntries;
    lpBitmapInfoHeader->biClrImportant = nColorTableEntries;

    DWORD dwBytes = ((DWORD) nWidth * nBitCount) / 32;
    if(((DWORD) nWidth * nBitCount) % 32) {
        dwBytes++;
    }
    dwBytes *= 4;

    DWORD dwSizeImage = dwBytes * nHeight;
    lpBitmapInfoHeader->biSizeImage = dwSizeImage;

    LPBYTE lpDibBits = 0;
    HBITMAP hBitmap = ::CreateDIBSection(hMemDC, (LPBITMAPINFO)lpBitmapInfoHeader, DIB_RGB_COLORS,  (void**)&lpDibBits, NULL, 0);
    SelectObject(hMemDC, hBitmap);
    BitBlt(hMemDC, 0, 0, nWidth, nHeight, hDC, 0, 0, SRCCOPY);
    ReleaseDC(hWnd, hDC);

    BITMAPFILEHEADER bmfh;
    bmfh.bfType = 0x4d42;  // 'BM'
    int nHeaderSize = sizeof(BITMAPINFOHEADER) + sizeof(RGBQUAD) * nColorTableEntries;
    bmfh.bfSize = 0;
    bmfh.bfReserved1 = bmfh.bfReserved2 = 0;
    bmfh.bfOffBits = sizeof(BITMAPFILEHEADER) + sizeof(BITMAPINFOHEADER) + sizeof(RGBQUAD) * nColorTableEntries;

    FILE *pFile = 0;
    pFile = fopen(szFilePath, "wb");
    if(!pFile)
    {
        ::DeleteObject(hBMP);
        ::DeleteObject(hBitmap);
        delete[]lpBitmapInfoHeader;
        cout << "can not open file\n";
        return false;
    }

    DWORD nColorTableSize = 0;
    if (nBitCount != 24)
        nColorTableSize = (1 << nBitCount) * sizeof(RGBQUAD);
    else
        nColorTableSize = 0;


    fwrite(&bmfh, sizeof(BITMAPFILEHEADER), 1, pFile);
    fwrite(lpBitmapInfoHeader, nHeaderSize,1,pFile);

    if(nBitCount < 16)
    {
        int nBytesWritten = 0;
        RGBQUAD *rgbTable = new RGBQUAD[nColorTableEntries * sizeof(RGBQUAD)];
        //fill RGBQUAD table and write it in file
        for(int i = 0; i < nColorTableEntries; ++i)
        {
            rgbTable[i].rgbRed = rgbTable[i].rgbGreen = rgbTable[i].rgbBlue = i;
            rgbTable[i].rgbReserved = 0;

            fwrite(&rgbTable[i], sizeof(RGBQUAD), 1, pFile);
        }
        delete[]rgbTable;

        /*
        RGBQUAD rgb;
        for (DWORD i = 0; i < nColorTableEntries ; i++)
        {
            rgb.rgbBlue = rgb.rgbGreen = rgb.rgbRed = (BYTE)(i*(255/(nColorTableEntries-1)));
            nBytesWritten = fwrite(&rgb, 1, sizeof(rgb), pFile);
            if (nBytesWritten != sizeof(rgb))
            {
                printf("error while writing rgb header\n");
                fclose(pFile);

                ::DeleteObject(hBMP);
                ::DeleteObject(hBitmap);
                delete[]lpBitmapInfoHeader;

                return false;
            }
        }
        */
    }

    fwrite(lpDibBits, dwSizeImage, 1, pFile);

    fclose(pFile);

    ::DeleteObject(hBMP);
    ::DeleteObject(hBitmap);
    delete[]lpBitmapInfoHeader;
}
//End TEMP


inline bool SameColours(COLORREF const &a, COLORREF const &b) {
    bool ret = true;

    ret &= GetRValue(a) == GetRValue(b);
    ret &= GetGValue(a) == GetGValue(b);
    ret &= GetBValue(a) == GetBValue(b);

    return ret;
}

class VirtualKeyboard
{
    COLORREF keyColUnpressed[N_PIANO_KEYS];
    unsigned int keyX[N_PIANO_KEYS]; //White then black, from left to right
    unsigned int keyY[N_PIANO_KEYS]; //White then black, from left to right

public:
    bool init(unsigned int xLeft, unsigned int yTop, unsigned int xRight, unsigned yBtm) {
        bool ret = true;

        //Calculate parameters of the virtual keyboard
        const float whtKeyHeight = (yBtm-yTop);
        const float whtKeyWidth = float(xRight-xLeft)/(N_WHITE_KEYS);

        //Calculate coordinates of the white keys
        for(unsigned int i = 0; i < N_WHITE_KEYS ; ++i) {
            keyX[i]=xLeft+(i+1.f/2)*float(whtKeyWidth);
            keyY[i]=yTop+3.f/4*float(whtKeyHeight);
        }

        //Calculate coordinates of the black keys
        unsigned int iBlkKey = 0;
        for(unsigned int i = 0 ; i < N_WHITE_KEYS-1 ; ++i) {
            //Determine if there is a black key
            bool skip = false;

            //Some black keys are absent from the offset white keys pattern - skip if applicable
            for(unsigned int j = 0 ; j < N_ABSENT_BLK_KEYS ; ++j) {
                if(i+1 == ABSENT_BLK_KEYS[j]) {
                    skip = true;
                    break;
                }
            }

            //If that key exists, add it to the list
            if(!skip) {
                keyX[iBlkKey+N_WHITE_KEYS]=xLeft+whtKeyWidth*(i+1);
                keyY[iBlkKey+N_WHITE_KEYS]=yTop+1.f/4*float(whtKeyHeight);
                iBlkKey++;
            }
        }

        //Capture the screen
        HDC hdcScreen = ::GetDC(GetDesktopWindow());
        captureAndSave(GetDesktopWindow(),32,"./capture.bmp");
        //And fill in the colors "at rest" for all the keys
        for(unsigned int i = 0 ; i < N_PIANO_KEYS ; ++i) {
            keyColUnpressed[i] = ::GetPixel(hdcScreen, keyX[i], keyY[i]);
            unsigned int r = GetRValue(keyColUnpressed[i]);
            unsigned int g = GetGValue(keyColUnpressed[i]);
            unsigned int b = GetBValue(keyColUnpressed[i]);

            ::SetPixel(hdcScreen, keyX[i], keyY[i], RGB(255,0,0)); //DEBUG: Breakpoint on this line, the RGB values are wrong for some keys (e.g. i=8 and all blacks)
            Sleep(100);
        }
        ReleaseDC(GetDesktopWindow(),hdcScreen);

        //Sanity check : all white keys should have the same colour, and all black keys their own colour as well
        for(unsigned int i = 1 ; i < N_PIANO_KEYS ; ++i) {
            if(i != 1 && i != N_WHITE_KEYS) {
                if(
                    !SameColours(
                                    keyColUnpressed[i],(i < N_WHITE_KEYS ? keyColUnpressed[0]
                                                                         : keyColUnpressed[N_WHITE_KEYS])
                                )
                  )
                {
                    ret = false;
                    break;
                }
            }
        }

        return 0;
    }
};


int main()
{
    VirtualKeyboard vk;

    cout << "You have 3 seconds to minimize this window and maximise MidiSheetMusic" << endl;
    Sleep(3000);

    cout << "Result of init() : " << vk.init(VKxLeft, VKyTop, VKxRight, VKyBtm) << endl;

    while(1)
    {
        //Get all keys pixels and reproduce on the LEDs on the physical keyboard
        Sleep(10); //Runs while not Ctrl+c
    }

    return 0;
}

Thank you very much in advance.

Swordfish
  • 12,971
  • 3
  • 21
  • 43
Mister Mystère
  • 952
  • 2
  • 16
  • 39
  • 1
    Why... are you taking a screenshot and calling `GetPixel` at all? Doesn't seem like there is any immediately obvious reason to do so... – IInspectable Nov 26 '18 at 16:13
  • GetPixel() is necessary to know the state of the key when the software plays a MIDI song, and screenshot is not part of the program, it was just to understand the results of GetPixel() – Mister Mystère Nov 26 '18 at 16:17
  • 1
    You are drawing and capturing the desktop window (`GetDesktopWindow()`), not the Window of MidiSheetMusic. – Swordfish Nov 26 '18 at 16:17
  • 2
    I think it would be much easier to just find a MIDI library and write a program that sets the LEDs from actual notes and timing than to have some 3rd party software draw a picture on the screen and try to interpret that. Also, `Get/SetPixel()` stuff is quite slow and (i think) way too error phrone for what you are trying to do. – Swordfish Nov 26 '18 at 16:18
  • This is a really roundabout way of achieving your goal. Just get the MIDI playback code and integrate with your code directly. MIDI playback isn't hard in and of itself, and solutions are widely available. – Bartek Banachewicz Nov 26 '18 at 16:22
  • Swordfish : captureAndSave() had that goal : check I was doing a GetPixel() on the correct picture - and it does get MidiSheetMusic. As for the method I used, I still think this is the easiest method for a non-pianist who never looked into MIDI files... If I hadn't encountered that problem, I'd already be done :/ – Mister Mystère Nov 26 '18 at 16:22
  • I understand this may not be your preferred way of achieving this goal, but could we focus on the problem at hand, even if it is for the sake of understanding, please? – Mister Mystère Nov 26 '18 at 16:23
  • 3
    @MisterMystère Then you should strip the code of everything music-related and just keep the problematic GetPixel part. If you don't want help with that part, then it's just introducing noise. – Bartek Banachewicz Nov 26 '18 at 16:25
  • Shooting a guess out there - Maybe use the function [DPtoLP](https://learn.microsoft.com/en-us/windows/desktop/api/wingdi/nf-wingdi-dptolp) on the key coordinates before calling GetPixel? – Govind Parmar Nov 26 '18 at 16:30
  • Wait. You are saying you got a hwnd and dc of the Desktop Window and what you copied was the stuff MidiSheetMusic was displaying? – Swordfish Nov 26 '18 at 16:36
  • @GovindParmar: Thanks, but didn't work :( – Mister Mystère Nov 26 '18 at 16:51
  • @Swordfish: yes, and I also replaced GetDesktopWindow() with NULL and still got an actual screen capture, and therefore the MidiSheetMusic interface – Mister Mystère Nov 26 '18 at 16:52
  • I just looked it up `GetDC()` gets a device context of the entire screen for `NULL`. You are right, I am wrong, Sowwy. – Swordfish Nov 26 '18 at 16:58
  • 1
    https://pastebin.com/ggCUiyav Works for me. Output: 302d2d abcdef. Of course not 100% reliable ... can be overwritten anytime ... but most times it works. So the problem must be somewhere else. Are you sure you are calculating the coordinates right? – Swordfish Nov 26 '18 at 17:08
  • Yes since I highlight the pixels in red woth setpixels at the same coordinates and I can see they are spot on... I have no idea why this did not work... But it worked and I don't think I have changed anything. So case closed even though not solved *shrug*. Sorry for the distraction... I am going to close this. Even though it ended up working, it was wayyyyyyy too slow though so I used GetDIBits instead. In case someone ends up in the same situation. – Mister Mystère Dec 04 '18 at 21:23

0 Answers0