Fliped cursor icon on desktop recording using directshow

Question

Using Directshow.NET I have developed an application which will record the desktop screen, to get mouse pointer we need to paint mouse pointer by our own. So I added SampleGrabber adn in BufferCB I have written below code:

    public const Int32 CURSOR_SHOWING = 0x00000001;

    [StructLayout(LayoutKind.Sequential)]
    public struct ICONINFO
    {
        public bool fIcon;
        public Int32 xHotspot;
        public Int32 yHotspot;
        public IntPtr hbmMask;
        public IntPtr hbmColor;
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct POINT
    {
        public Int32 x;
        public Int32 y;
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct CURSORINFO
    {
        public Int32 cbSize;
        public Int32 flags;
        public IntPtr hCursor;
        public POINT ptScreenPos;
    }

    [DllImport("user32.dll")]
    public static extern bool GetCursorInfo(out CURSORINFO pci);

    [DllImport("user32.dll")]
    public static extern IntPtr CopyIcon(IntPtr hIcon);

    [DllImport("user32.dll")]
    public static extern bool DrawIcon(IntPtr hdc, int x, int y, IntPtr hIcon);

    [DllImport("user32.dll")]
    public static extern bool GetIconInfo(IntPtr hIcon, out ICONINFO piconinfo);

public int BufferCB(double SampleTime, IntPtr pBuffer, int BufferLen)
{
Graphics g;
Bitmap v;
v = new Bitmap(m_videoWidth, m_videoHeight, m_stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, pBuffer);            
g = Graphics.FromImage(v);
CURSORINFO cursorInfo;
cursorInfo.cbSize = Marshal.SizeOf(typeof(CURSORINFO));

if (GetCursorInfo(out cursorInfo))
{
    if (cursorInfo.flags == CURSOR_SHOWING)
    {
        var iconPointer = CopyIcon(cursorInfo.hCursor);
        ICONINFO iconInfo;
        int iconX, iconY;

        if (GetIconInfo(iconPointer, out iconInfo))
        {
            iconX = cursorInfo.ptScreenPos.x - ((int)iconInfo.xHotspot);
            iconY = cursorInfo.ptScreenPos.y - ((int)iconInfo.yHotspot);

            DrawIcon(g.GetHdc(), iconX, iconY, cursorInfo.hCursor);
            g.ReleaseHdc();
            g.Dispose();
            v.Dispose();                        
        }
    }
}
return 0;
}

This code is painting the mouse cursor but cursor is flipped on Y axis.

It's may be because in BufferCB if we convert pBuffer in Bitmap then that frame is flipped on Y axis. To solve this I flipped current frame on Y axis by adding v.RotateFlip(RotateFlipType.RotateNoneFlipY); inside BufferCB after this change mouse pointer is not visible in desktop recording video.

How I can flip mouse pointer?

Update #1

I converted icon pointer to Icon then into Bitmap using Icon.ToBitmap() and then flipped on Y axis, here is code (Thanks to @Roman R.):

...
iconX = cursorInfo.ptScreenPos.x - ((int)iconInfo.xHotspot);
iconY = cursorInfo.ptScreenPos.y - ((int)iconInfo.yHotspot);

Icon ic = Icon.FromHandle(iconPointer);
Bitmap icon = ic.ToBitmap();
icon.RotateFlip(RotateFlipType.RotateNoneFlipY);
g.DrawImage(icon, iconX, iconY);
g.Dispose();
v.Dispose();
icon.Dispose();
ic.Dispose();  
...

Only one issue I am facing in above modification sometimes I get ArgumentException at line Bitmap icon = ic.ToBitmap();

  System.ArgumentException occurred
  HResult=-2147024809
  Message=Parameter is not valid.
  Source=System.Drawing
  StackTrace:
       at System.Drawing.Bitmap.FromHicon(IntPtr hicon)
  InnerException:

I disposed all the bitmaps used, still i get this exception.

One option that I tried is flipping cursor image by traversing in reverse order on `cursorInfo.hCursor` and then pass new `IntPtr` to `DrawIcon`..but I feel this is a very huge overhead as `BufferCB` is getting called on every frame. — Amogh, Feb 19 '16 at 21:19

score 1 · Answer 1 · answered Feb 19 '16 at 21:51

You have another option of painting correct mouse pointer. Just take a mouse pointer's PNG image and in BufferCB:

public int BufferCB(double SampleTime, IntPtr pBuffer, int BufferLen)
{
Graphics g;
Bitmap v;
v = new Bitmap(m_videoWidth, m_videoHeight, m_stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, pBuffer);            
g = Graphics.FromImage(v);
CURSORINFO cursorInfo;
cursorInfo.cbSize = Marshal.SizeOf(typeof(CURSORINFO));

if (GetCursorInfo(out cursorInfo))
{
    if (cursorInfo.flags == CURSOR_SHOWING)
    {
        var iconPointer = CopyIcon(cursorInfo.hCursor);
        ICONINFO iconInfo;
        int iconX, iconY;

        if (GetIconInfo(iconPointer, out iconInfo))
        {
            iconX = cursorInfo.ptScreenPos.x - ((int)iconInfo.xHotspot);
            iconY = cursorInfo.ptScreenPos.y - ((int)iconInfo.yHotspot);

            //DRAW STATIC POINTER IMAGE
            Bitmap pointerImage = new Bitmap('pointer.png');
            g.DrawImage(pointerImage,iconX,iconY);

            g.Dispose();
            v.Dispose();                        
        }
    }
}
return 0;
}

I think `pointer.png` is need to be already flipped on Y axis, so that it will be get correctly painted. — user3813463, Feb 19 '16 at 21:52
Yes, I already tried this approach and it's working fine. Why I want Win32 API based solution is because I want actual cursor image in video. For example if you open Excel then while working inside worksheet a different icon ([plus sign icon](http://ckjh.cksd.wednet.edu/staff/charlesp/Excel/img15.gif)) will come, so if I implement this solution than only normal pointer will get painted in video. — Amogh, Feb 19 '16 at 21:55

Roman R. · Accepted Answer · 2016-02-23T11:13:30.780

1

The problem you have here is your BufferCB implementation. You create a temporary Graphics/Bitmap object, so that it could be updated with the cursor overlay. You have this bitmap upside down and normal draw of the icon applies it flipped.

You need to take into account that normal order of lines of 24-bit RGB buffer you get with BufferCB callback is bottom-to-top; order of lines Bitmap constructor expects is the opposite. You need to transfer lines into Bitmap in reversed order, then get them back respectively. I am not sure if negative stride works out there well, presumably it will not. Or simply overlay the cursor pre-flipped, to compensate for flipped background and overlay coordinates too.

edited Feb 23 '16 at 11:13

answered Feb 23 '16 at 11:06

Roman R.

68,205
6
94
158

If i flip temporary bitmap created in `BufferCB` using code `v.RotateFlip(RotateFlipType.RotateNoneFlipY);` then also its not working, why? – Amogh Feb 23 '16 at 11:10
1

Because your code runs in assumption that results are transferred back to `pBuffer`, supposedly your flipping detaches bitmap from original data buffer. – Roman R. Feb 23 '16 at 11:12
Oh!! So what if after `DrawIcon` I again flip the `Bitmap` so that it will be bottom at top with overlay cursor and then re-assign `pBuffer` with new `Bitmap`, will it be work? what i feel is transferring lines into `Bitmap` in reversed order and flipping `Bitmap` is the same process, Is it? – Amogh Feb 23 '16 at 11:18
I tried with second option "pre-flipped cursor", actually I am not able to generate per-flipped cursor by `cursorInfo.hCursor`. – Amogh Feb 23 '16 at 11:20
1

You choose either of the two. I'd perhaps go with transferring line by line to get bitmap in correct orientation. To pre-flip cursor, draw it into worker bitmap, flip it, then apply it on your main bitmap. – Roman R. Feb 23 '16 at 11:21
Okay :) so i will try by both. flipping and re-flipping is easy. but according to suggestion by you I will write a `Bitmap` line by line by reversing `pBuffer` put a overlay then again again give it back to `pBuffer` in reverse order. – Amogh Feb 23 '16 at 11:28
I tried the same for per-flipped cursor icon, but by doing that transparency of cursor icon is getting lost. – Amogh Feb 23 '16 at 11:28
Currently code is not with me, I will try it tonight and will update the same. Thanks for your guidance. – Amogh Feb 23 '16 at 11:30
I implemented the the solution as per your second suggestion, flipping of icon itself. I added code Update #1 in question. I am still in search of reversing `pBuffer` I tried a lot but no success. – Amogh Feb 25 '16 at 15:03
Create empty `Bitmap` (of compatible format - what you already do) and use [`LockBits`](https://msdn.microsoft.com/en-us/library/5ey6h79d), `UnlockBits` to access individual lines and transfer data back and forth between media sample buffer and bitmap buffer. – Roman R. Feb 25 '16 at 15:18

Fliped cursor icon on desktop recording using directshow

2 Answers2

Linked