C# Screen streaming program

Question

Lately, I have been working on a simple screen-sharing program.

Actually, the program works on a TCP protocol and uses the Desktop duplication API- a cool service that supports very fast screen capturing and also provides information about MovedRegions(areas that only changed their position on the screen but still exist) and UpdatedRegions(changed areas).

The Desktop duplication has 2 important properties-2 byte arrays an array for the previous-pixels and a NewPixels array. Every 4 bytes represent a pixel in the RGBA form so for example if my screen is 1920 x 1080 the buffer size is 1920 x 1080 * 4.

Below are the important highlights of my strategy

1. In the initial state (the first time) I send the entire pixel buffer (in my case it's 1920 x 1080 * 3) - the alpha component is always 255 on screens :)
2. From now on, I iterate over the UpdatedRegions (it's a rectangles array) and I send the regions bounds and Xo'r the pixels in it something like this:

writer.Position = 0;
var n = frame._newPixels;
var w = 1920 * 4; //frame boundaries.
var p = frame._previousPixels;

foreach (var region in frame.UpdatedRegions)
{
    writer.WriteInt(region.Top);
    writer.WriteInt(region.Height);
    writer.WriteInt(region.Left);
    writer.WriteInt(region.Width);
    
    for (int y = region.Top, yOffset = y * w; y < region.Bottom; y++, yOffset += w)
    {
        for (int x = region.Left, xOffset = x * 4, i = yOffset + xOffset; x < region.Right; x++, i += 4)
        {
            writer.WriteByte(n[i] ^ p[i]); //'n' is the newpixels buffer and 'p' is the previous.xoring for differences.
            writer.WriteByte(n[i+1] ^ p[i+1]);
            writer.WriteByte(n[i + 2] ^ p[i + 2]);

        }
    }
}

1. I Compress the buffer using the lz4 wrapper written in c# (refer to lz4.NET. Then, I write the data on a NetworkStream.
2. I merge the areas on the receiver side to get the updated image - this is not our problem today :)

'writer' is an instance of the 'QuickBinaryWriter' class I wrote (simply to reuse the same buffer again).

public class QuickBinaryWriter
{
    private readonly byte[] _buffer;
    private int _position;

    public QuickBinaryWriter(byte[] buffer)
    {
        _buffer = buffer;
    }

    public int Position
    {
        get { return _position; }
        set { _position = value; }
    }

    public void WriteByte(byte value)
    {
        _buffer[_position++] = value;
    }


    public void WriteInt(int value)
    {
        byte[] arr = BitConverter.GetBytes(value);
        
        for (int i = 0; i < arr.Length; i++)
            WriteByte(arr[i]);
    }
   
}

From many measures, I've seen that the data sent is really huge, and sometimes for a single frame update the data could get up to 200kb (after compression!). Let's be honest-200kb is really nothing, but if I want to stream the screen smoothly and watch at a high Fps rate I would have to work on this a little bit - to minimize the network traffic and bandwidth usage.

I'm looking for suggestions and creative ideas to improve the efficiency of the program- mainly the data sent on the network part (by packing it in other ways or any other idea) I'll appreciate any help and ideas. Thanks!

Answer 1

For your screen of 1920 x 1080, with 4 byte color, you are looking at approximately 8 MB per frame. With 20 FPS, you have 160 MB/s. So getting from 8 MB to 200 KB (4 MB/s @ 20 FPS) is a great improvement.

I would like to get your attention to certain aspects that I am not sure you are focusing on, and hopefully it helps.

1. The more you compress your screen image, the more processing it might need
2. You actually need to focus on compression mechanisms designed for series of continuously changing images, similar to video codecs (sans audio though). For example: H.264
3. Remember, you need to use some kind of real-time protocol for transferring your data. The idea behind that is, if one of your frame makes it to the destination machine with a lag, you might as well drop the next few frames to play catch-up. Else you will be in a perennially lagging situation, which I doubt the users are going to enjoy.
4. You can always sacrifice quality for performance. The simplest such mechanism that you see in similar technologies (like MS remote desktop, VNC, etc) is to send a 8 bit color (ARGB each of 2 bits) instead of 3 byte color that you are using.
5. Another way to improve your situation would be to focus on a specific rectangle on the screen that you want to stream, instead of streaming the whole desktop. This will reduce the size of the frame itself.
6. Another way would be to scale your screen image to a smaller image before transmitting and then scale it back to normal before displaying.
7. After sending the initial screen, you can always send the diff between newpixels and previouspixels. Needless to say the the original screen and the diff screen will all be LZ4 compressed/decompressed. Every so often you should send the full array instead of the diff, if you use some lossy algorithm to compress the diff.
8. Does UpdatedRegions, have overlapping areas? Can that be optimized to not send duplicate pixel information?

The ideas above can be applied one on top of the other to get a better user experience. Ultimately, it depends on the specifics of your application and end-users.

EDIT:

• Color Quantization can be used to reduce the number of bits used for a color. Below are some links to concrete implementations of Color Quantization

  • Optimizing Color Quantization for Images
  • nQuant library

• Usually the quantized colors are stored in a Color Palette and only the index into this palette is given to the decoding logic

Answer 2

Slashy,

Since you are using a high res frames and you want a good frame rate you're likely going to be looking at H.264 encoding. I've done some work in HD/SDI broadcast video which is totaly dependent on H.264, and a little now moving to H.265. Most of the libraries used in broadcast are written in C++ for speed.

I'd suggest looking at something like this https://msdn.microsoft.com/en-us/library/windows/desktop/dd797816(v=vs.85).aspx