3

I am working on screen sharing project. I am sending only screen differences over socket comparing previous and actual buffer. It working

I am sending 8 to 9 FPS to client using Format16bppRgb555 to reduce overall bytes size of Bitmap

byte[] wholescreensize= new byte[1360 * 768 * 2];// Its around 2 Mb

My problem Is when full screen is changed.

I am getting about 45-60 kb of PNG image using below function

45kb * 10 (FPS) = 450 kb

It is possible to reduce beyond 45 kb.

I am not interested to reduce FPS as it live screen sharing app.

JPEG Compression or LZ4/GZIP also not making much difference as PNG image already compressed

private void SendImgDiffToClient(byte[] contents,Rectangle rectangle)
{   

    //Converting Small Portion to Bitmap.Bcoz Image.FromStrem not working here error Parameter is not Valid
    byte[] byteArrayout = new byte[contents.Length];

    var bitmap = new Bitmap(rectangle.Width, rectangle.Height, PixelFormat.Format16bppRgb555);
    var bitmap_data = bitmap.LockBits(new Rectangle(0, 0, bitmap.Width, bitmap.Height), ImageLockMode.WriteOnly, PixelFormat.Format16bppRgb555);
    Marshal.Copy(contents, 0, bitmap_data.Scan0, byteArrayout.Length);
    bitmap.UnlockBits(bitmap_data);

    //Converting Small Bitmap to Png Byte Array and Sending to Client
    using (MemoryStream ms = new MemoryStream())
    { 
        Image msImage = (Image)bitmap;
        msImage.Save(ms, ImageFormat.Png);

        msImage.Dispose();
        byteArrayout = ms.ToArray();
    }  

    SendtoClient(byteArrayout);
}

My Questing is what is a best approach to reduce bytes in such scenario.

Azar Shaikh
  • 445
  • 9
  • 26
  • PNG is lossless compression. If you want to reduce it even further, use lossy (JPG) compression, inter-frame compression (mpg) and/or lower the framerate. – Manfred Radlwimmer Mar 05 '18 at 10:25
  • 1
    There is no good way to do this, always too slow for modern monitor sizes. Use the built-in OS support for this, google "c# remote desktop services" to find hits. It gets its speed by encoding video driver commands instead of pixels. – Hans Passant Mar 05 '18 at 10:31
  • @HansPassant i am already researching for this. i am not getting any appropriate solution for that. tried this but no luck https://stackoverflow.com/questions/31543940/c-sharp-screen-transfer-over-socket-efficient-improve-ways – Azar Shaikh Mar 05 '18 at 10:34
  • 2
    Well, I told you so. Use RDP. – Hans Passant Mar 05 '18 at 10:36
  • i am using vnc protocol here. unfortunately i cant switch. Client communicating with vnc protocol – Azar Shaikh Mar 05 '18 at 10:39
  • Your going to run into two issues here. If you process the images to make them smaller, you will run into sluggish image updates due to the client side processing. If you Send the unprocessed images to the viewer, then they will encounter network sluggishness. Do the images need to be in real-time? How about a few seconds lag? – Baaleos Mar 05 '18 at 10:41
  • @Baaleos Its ok with few seconds lag but unfortunately i am not getting any solution – Azar Shaikh Mar 05 '18 at 10:43
  • try using base64 for sending it. that might shrink the size – slow Mar 05 '18 at 10:50
  • @sLowDowN it increasing bytes size – Azar Shaikh Mar 05 '18 at 10:56
  • You might be able to find a compromise between speed and bandwidth by chunking your frame data. Eg: Build up 10 seconds of frame data - process it into an MP4 or AVI with good compression - send it to the client. Have the client save that 10 seconds of buffered data, and then play it back. Processing individual frames is not going to really work for sending frame differences - as you need to have the previous frame in memory to calculate 'the differences'. If you use a video codec to process a few seconds of frames, you can probably save yourself issues associated with frame processing – Baaleos Mar 05 '18 at 16:54
  • When full screen is changed, you should no longer chunk the data, just send the full image. Also, you could reduce chunk sizes by using .PixelData(if I remember correctlly the name, without headers and all that Jazz. which would also reduce the time to reassemble the image, since it would already be in a better format to “paste” in the right places, strip by strip. – Alexandru Clonțea Mar 07 '18 at 11:31
  • Actually the little cubits result, namely the pixel data. Also make sure you are not using a format that uses an alpha channel! – Alexandru Clonțea Mar 07 '18 at 11:38
  • Little cubits == Lockbits autocorrect (mobile). Compression could be nonimage compression maybe, you should send diff,, but need some sort of priority queue to invalidate region render to latest version... stream of thought here, I will take a closer look tonight/weekend. Hans Passant has the best advice here, try to use ready-built, tough subject! – Alexandru Clonțea Mar 07 '18 at 11:52
  • Thanks @AlexandruClonțea – Azar Shaikh Mar 07 '18 at 11:54
  • https://github.com/humphd/VncSharp – Alexandru Clonțea Mar 07 '18 at 12:37
  • @AlexandruClonțea vncsharp is vnc client side implementation. I am getting issue on vnc server side to send image difference buffer update – Azar Shaikh Mar 07 '18 at 13:06
  • 3
    Then, https://github.com/T1T4N/NVNC could inspire you :) – Alexandru Clonțea Mar 07 '18 at 13:36
  • 1. use UDP for that purposes 2. you could try to either one of two choises - calculate difference with your own code and send only diffs which are zipped - you could try to send h.264 video stream (which will calculate diff, compress and so on while decoding). – Igor Gnedysh Mar 13 '18 at 09:50
  • MJPEGStream is the answer – Mauro Sampietro Mar 13 '18 at 15:39
  • I've done this using DirectX and Win32 sdk calls in C#. I was able to capture well above the 10 FPS. I suggest some searches on those keywords. This article is using C. You can easily marshall the calls for C#. https://www.codeproject.com/Articles/5051/Various-methods-for-capturing-the-screen – tatmanblue Mar 13 '18 at 19:43

1 Answers1

2

Video streaming is essentially what you're doing; and modern video compression algorithms have lots of enhancements. Perhaps they can track or move an artifact, or otherwise distort said artifact as part of their functionality. Perhaps they can stream the data in a progressively building manner, so that static items eventually acquire more detail (similar to progressive jpeg images.) They do lots of things all at the same time. You can try to research them further, and take inspiration from them, or you could pick and use one.

This is to say that many people here seem to prefer the solution of using a readily available video compression library. Especially if you are worried about streaming bandwidth.

If you don't want to use an existing video library, then you have to decide how much effort you want to put in, versus how sloppy you want to be with consuming more bandwidth than otherwise necessary.

Greg
  • 2,410
  • 21
  • 26