Android TextureView / Drawing / Painting Performance

Question

I am attempting to make a drawing/painting app using TextureView on Android. I want to support a drawing surface of up to 4096x4096 pixels, which seems reasonable for my minimum target device (which I use for testing) which is a Google Nexus 7 2013 which has a nice quad core CPU and 2GB memory.

One of my requirements is that my view must be inside of a view that allows it to be zoomed in and out and panned, which is all custom code I have written (think UIScrollView from iOS).

I've tried using a regular View (not TextureView) with OnDraw and performance was absolutely horrible - less than 1 frame a second. This happened even if I called Invalidate(rect) with only the rect that changed. I tried turning off hardware acceleration for the view, but nothing rendered, I assume because 4096x4096 is too big for software.

I then tried using TextureView and performance is a little better - about 5-10 frames per second (still terrible but better). The user draws into a bitmap, which is then later drawn into the texture using a background thread. I'm using Xamarin but hopefully the code makes sense to Java people.

private void RunUpdateThread()
{
    try
    {
        TimeSpan sleep = TimeSpan.FromSeconds(1.0f / 60.0f);
        while (true)
        {
            lock (dirtyRect)
            {
                if (dirtyRect.Width() > 0 && dirtyRect.Height() > 0)
                {
                    Canvas c = LockCanvas(dirtyRect);
                    if (c != null)
                    {
                        c.DrawBitmap(bitmap, dirtyRect, dirtyRect, bufferPaint);
                        dirtyRect.Set(0, 0, 0, 0);
                        UnlockCanvasAndPost(c);
                    }
                }
            }
            Thread.Sleep(sleep);
        }
    }
    catch
    {
    }
}

If I change lockCanvas to pass null instead of a rect, performance is great at 60 fps, but the contents of the TextureView flicker and get corrupted, which is disappointing. I would have thought it would simply be using an OpenGL frame buffer / render texture underneath or at least have an option to preserve contents.

Are there any other options short of doing everything in raw OpenGL in Android for high performance drawing and painting on a surface that is preserved in between draw calls?

Answer 1

First off, if you want to understand what's going on under the hood, you need to read the Android Graphics Architecture document. It's long, but if you sincerely want to understand the "why", it's the place to start.

About TextureView

TextureView works like this: it has a Surface, which is a queue of buffers with a producer-consumer relationship. If you're using software (Canvas) rendering, you lock the Surface, which gives you a buffer; you draw on it; then you unlock the Surface, which sends the buffer to the consumer. The consumer in this case is in the same process, and is called SurfaceTexture or (internally, more aptly) GLConsumer. It converts the buffer into an OpenGL ES texture, which is then rendered to the View.

If you turn off hardware acceleration, GLES is disabled, and TextureView cannot do anything. This is why you got nothing when you turned hardware acceleration off. The documentation is very specific: "TextureView can only be used in a hardware accelerated window. When rendered in software, TextureView will draw nothing."

If you specify a dirty rect, the software renderer will memcpy the previous contents into the frame after rendering is complete. I don't believe it sets a clip rect, so if you call drawColor(), you will fill the entire screen, and then have those pixels overwritten. If you aren't currently setting a clip rect, you may see some performance benefit from doing so. (I didn't check the code though.)

The dirty rect is an in-out parameter. You pass the rect you want in when you call lockCanvas(), and the system is allowed to modify it before the call returns. (In practice, the only reason it would do this would be if there were no previous frame or the Surface were resized, in which case it would expand it to cover the entire screen. I think this would have been better handled with a more direct "I reject your rect" signal.) You're required to update every pixel inside the rect you get back. You are not allowed to alter the rect, which you appear to be trying to do in your sample -- whatever is in the dirty rect after lockCanvas() succeeds is what you're required to draw on.

I suspect the dirty rect mis-handling is the source of your flickering. Sadly, this is an easy mistake to make, as the behavior of the lockCanvas() dirtyRect arg is only documented in the Surface class itself.

Surfaces and buffering

All Surfaces are double- or triple-buffered. There is no way around this -- you cannot read and write simultaneously and not get tearing. If you want a single buffer that you can modify and push when desired, that buffer will need to be locked, copied, and unlocked, which creates stalls in the composition pipeline. For best throughput and latency, flipping buffers is better.

If you want the lock-copy-unlock behavior, you can write that yourself (or find a library that does it), and it will be as efficient as it would be if the system did it for you (assuming you're good with blit loops). Draw to an off-screen Canvas and blit the bitmap, or to a OpenGL ES FBO and blit the buffer. You can find an example of the latter in Grafika's "record GL app" Activity, which has a mode that renders once off-screen, and then blits twice (once for display, once for recording video).

More speed and such

There are two basic ways to draw pixels on Android: with Canvas, or with OpenGL. Canvas rendering to a Surface or Bitmap is always done in software, while OpenGL rendering is done with the GPU. The only exception is that, when rendering to a custom View, you can opt to use hardware acceleration, but this does not apply when rendering to the Surface of a SurfaceView or TextureView.

A drawing or painting app can either remember the drawing commands, or just throw pixels at a buffer and use that as its memory. The former allows for deeper "undo", the latter is much simpler, and has increasingly better performance as the amount of stuff to render grows. It sounds like you want to do the latter, so blitting from off-screen makes sense.

Most mobile devices have a hardware limitation of 4096x4096 or smaller for GLES textures, so you won't be able to use a single texture for anything larger. You can query the size limit value (GL_MAX_TEXTURE_SIZE), but you may be better off with an internal buffer that is as large as you want, and just render the portion that fits on screen. I don't know what the Skia (Canvas) limitation is offhand, but I believe you can create much larger Bitmaps.

Depending on your needs, a SurfaceView may be preferable to a TextureView, as it avoids the intermediate GLES texture step. Anything you draw on the Surface goes directly to the system compositor (SurfaceFlinger). The down side to this approach is that, because the Surface's consumer is not in-process, there is no opportunity for the View system to handle the output, so the Surface is an independent layer. (For a drawing program this could be beneficial -- the image being drawn is on one layer, your UI is on a separate layer on top.)

FWIW, I haven't looked at the code, but Dan Sandler's Markers app might be worth a peek (source code here).

Update: the corruption was identified as a bug and fixed in 'L'.

Answer 2

UPDATE I ditched TextureView and now use an OpenGL view where I call glTexSubImage2D to update changed pieces of a render texture.

OLD ANSWER I ended up tiling TextureView in a 4x4 grid. Depending on the dirty rect each frame, I refresh the appropriate TextureView views. Any view that is not updated that frame I call Invalidate on.

Some devices, such as the Moto G phone have an issue where the double buffering is corrupted for one frame. You can fix that by calling lockCanvas twice when the parent view has it's onLayout called.

private void InvalidateRect(int l, int t, int r, int b)
{
    dirtyRect.Set(l, t, r, b);

    foreach (DrawSubView v in drawViews)
    {
        if (Rect.Intersects(dirtyRect, v.Clip))
        {
            v.RedrawAsync();
        }
        else
        {
            v.Invalidate();
        }
    }

    Invalidate();
}

protected override void OnLayout(bool changed, int l, int t, int r, int b)
{
    for (int i = 0; i < ChildCount; i++)
    {
        View v = GetChildAt(i);
        v.Layout(v.Left, v.Top, v.Right, v.Bottom);
        DrawSubView sv = v as DrawSubView;
        if (sv != null)
        {
            sv.RedrawAsync();

            // why are we re-drawing you ask? because of double buffering bugs in Android :)
            PostDelayed(() => sv.RedrawAsync(), 50);
        }
    }
}