Improve rendering performance of animated image in browser

Question

I'm experimenting with different ways of rendering image in a browser. The idea is to create custom HTML element capable of displaying a set of images or frames as a continuous animation. The images will be generated on the server side and streamed to the browser client. The easiest way seems to be using img tag consistently updates its src attribute. For this, I subscribe to onload event and once image is loaded, I update image URL with random timestamp and it automatically sends new request creating infinite loop that looks like an animation. The solution is pretty simple and working, but affects CPU performance and doesn't use any GPU acceleration.

Controller used as an image source

[ApiController]
public class StreamController : ControllerBase
{
  [Route("/source")]
  public ActionResult Get()
  {
    var o = new Random();
    var pos = o.Next(50, 150);
    var map = new SKBitmap(250, 250);
    var canvas = new SKCanvas(map);
    var paint = new SKPaint
    {
      Color = SKColors.Black,
      Style = SKPaintStyle.Fill,
      FilterQuality = SKFilterQuality.High
    };

    canvas.DrawCircle(pos, pos, 20, paint);

    var image = SKImage.FromBitmap(map);
    var ms = new MemoryStream();

    image.Encode(SKEncodedImageFormat.Png, 100).SaveTo(ms);
    ms.Position = 0;

    return new FileStreamResult(ms, "image/png");
  }
}

HTML page

@page "/"

<!-- Solution #1 : SRC attribute -->

<img 
  width="250" 
  height="250" 
  @onload="OnLoad" 
  src="@Source" />

<!-- Solution #2 : background style URL -->

<div 
  style="width: 250px; height: 250px; background: url(@Source)">
</div>

<!-- Solution #3 : SRC of the picture element - does not work -->

<picture style="width: 250px; height: 250px">
  <source srcset="@Source" type="image/png" media="(min-width:250px)">
</picture>

@code 
{
  private Random _generator = new();
  public string Source { get; set; } = "/source";

  public void OnLoad()
  {
    // Animate by creating infinite loop of HTTP calls 
    // updating image source right after loading the previous one

    var uid = Guid.NewGuid().ToString("N");
    var num = _generator.Next();
    var stamp = DateTime.Now.Ticks;
    
    Source = $"/source?{ uid }-{ num }-{ stamp }";
  }
}

The outcome

Red border is HTML animation. Blue border is CSS background. Picture tag didn't render.

https://youtu.be/gk8Z-LrKxLE

The questions

1. Why solution #3 using picture element isn't working?
2. Why solution #2 with CSS background is much slower than solution #1 with image tag. Why is it skipping some frames and is not GPU accelerated?
3. Is there a way to decrease load on CPU by changing something in HTML or in the controller, e.g. switching to async streams in controller or convert images to a video stream?

Update

Appeared to be there is much bigger issue with refreshing img URL. Looks like FileStreamResult returned from ASP controller gets locked by an image, so every time I request an image update, like

• /source?1
• /source/2
• /source?3

...browser creates and caches this image and .NET can't release this resource which leads to enormous memory increase.

Answer 1

Found a solution - using Motion JPEG as a source for the img or iframe tag. Using video tag should be also be possible. Appeared to be most of browsers, if not all of them, support a stream of images sent via HTTP as multipart/mixed content where each image in this stream is just a set of bytes separated by some separator.

Motion JPEG Format

Sending the following via HTTP will make browser waiting for continuous data from the stream.

HTTP/1.1 200 OK\r\n
Content-Type: multipart/x-mixed-replace; boundary=--SomeBoundary\r\n

The server side producing images should keep the stream open, e.g. using infinite loop. This loop will be producing new images, converting them to appropriate format like JPEG, or even better optimized WEBP, and sending them to the HTTP stream as a set of bytes.

--SomeBoundary\r\n
Content-Type: image/jpeg\r\n
Content-Length: 100\r\n\r\n 
<Image-Bytes-Go-Here>
\r\n

As a result, to show a video that consists of 3 frames, the resulting HTTP response would look like this.

HTTP/1.1 200 OK\r\n
Content-Type: multipart/x-mixed-replace; boundary=--SomeBoundary\r\n

--SomeBoundary\r\n
Content-Type: image/jpeg\r\n
Content-Length: 100\r\n\r\n 
<Image-Bytes-Go-Here>\r\n

--SomeBoundary\r\n
Content-Type: image/jpeg\r\n
Content-Length: 100\r\n\r\n 
<Image-Bytes-Go-Here>\r\n

--SomeBoundary\r\n
Content-Type: image/jpeg\r\n
Content-Length: 100\r\n\r\n 
<Image-Bytes-Go-Here>\r\n

Optimization

• Simple shapes could be drawn with pure CSS
• Rendering of the main view with lots of dynamic shapes is better done with SkiaSharp canvas
• Text rendering in SkiSharp is somewhat slow. SVG or plain HTML may be the best option here

Memory leaks

The next issue is consistently increasing amount of memory when HTTP stream with images is being requested by img tag via HTTP.

<img src="http://127.0.0.1/stream" />

Surprisingly, simply replacing img tag with iframe solves the issue with memory leaks, most probably because frames and browser windows don't cache images as aggressively as img tag or at least this cache is being cleaned up more efficiently.

<iframe src="http://127.0.0.1/stream" />

Links explaining this concept with examples.

• https://en.wikipedia.org/wiki/Motion_JPEG
• https://blog.green.web.za/2019/11/23/mjpeg-in-asp-net-core.html
• https://www.codeproject.com/Articles/371955/Motion-JPEG-Streaming-Server

Real-time charting app implemented as reusable Blazor control.

• https://github.com/Indemos/Canvas-V3