Z-order issue with MediaCodec and TextureView

Question

In my Android app I have the need to render three views with the following Z-order:

1. At bottom, the output surface of a MediaCodec decoder covering the whole screen. I have the requirement that I have to transform the image produced by MediaCodec (e.g. scale it)
2. In the middle, a GLSurfaceView (or other surface/view running GL shaders I define), covering the whole screen. Obviously some of the pixels in this layer will be transparent, in order to see the MediaCodec output beneath.
3. On top, any other view - say an ImageView. Not sure if I will require transparency for these topmost views, maybe fully opaque rectangular Views are OK - they just won't cover the whole screen and will move around.

It's looking like this is impossible but maybe I'm missing something or there is a way to do it with more effort at a lower level (e.g. EGL contexts or something like that...which I currently don't understand).

The reason I haven't been able to get this to work and am worrying it is impossible is:

• For the bottom MediaCodec output layer (1), I have to be able to transform the image. So, the surface I give MediaCodec to render to must be from a TextureView
• In order to be able to see through the transparent pixels of the middle GLSurfaceView (2), I have to call GLSurfaceView.setZOrderOnTop(true). Otherwise the GLSurfaceView is opaque.
• But calling GLSurfaceView.setZOrderOnTop(true) means that no other views (3) are rendered on top of the GLSurfaceView. E.g. an ImageView will always appear behind the opaque pixels of the GLSurfaceView.

It looks like calling GLSurfaceView.setZOrderMediaOverlay(true) instead of GLSurfaceView.setZOrderOnTop(true) is intended to address this and facilitate this type of Z-ordering. And it does if the bottommost MediaCodec output layer is a SurfaceView. But I need it to be a TextureView so I can transform it. And GLSurfaceView.setZOrderMediaOverlay(true) doesn't seem to work when there is a TextureView beneath it: the TextureView is completely obscured by the middle GLSurfaceView layer rather than showing through the transparent pixels.

Is it correct that this Z-ordering is impossible? Or can it be accomplished by messing around with EGL and contexts, etc?

Answer 1

EGL contexts aren't really relevant here. Your fight is with SurfaceFlinger and the view system.

If you run adb shell dumpsys SurfaceFlinger you can see something a complete list of all the layers that the system compositor knows about. If you're playing a 320x240 video in a SurfaceView, it looks something like this (several columns and lots of other stuff removed for brevity):

    type    |          source crop              |           frame           name 
------------+-----------------------------------+--------------------------------
        HWC | [    0.0,    0.0,  320.0,  240.0] | [   48,  411, 1032, 1149] SurfaceView
        HWC | [    0.0,   75.0, 1080.0, 1776.0] | [    0,   75, 1080, 1776] com.android.grafika/com.android.grafika.PlayMovieSurfaceActivity
        HWC | [    0.0,    0.0, 1080.0,   75.0] | [    0,    0, 1080,   75] StatusBar
        HWC | [    0.0,    0.0, 1080.0,  144.0] | [    0, 1776, 1080, 1920] NavigationBar
  FB TARGET | [    0.0,    0.0, 1080.0, 1920.0] | [    0,    0, 1080, 1920] HWC_FRAMEBUFFER_TARGET

The layers are in Z-order, from back to front. The SurfaceView's surface layer is at the back, the app UI layer is on top of that, and the system status + nav bars are above everything.

Everything in the app's View hierarchy is rendered on a single layer. That includes TextureView. You can't control its Z-ordering relative to other hardware-composited layers.

SurfaceViewis fancy in that the View portion is just a transparent placeholder, while the real action happens on that separate layer, whose Z-ordering you can control (a little). You can put it on three different levels:

• "media" (the default)
• "media overlay"
• (app UI is here)
• "panel" (ZOrderOnTop)

So what you want to do is put your MediaCodec output at the default layer, and your GLES output at the "media overlay" layer. You need to do both of these with a SurfaceView.

It's hard to provide better advice from here because you described the problems you're having with your attempted solution, rather than the problem you're trying to solve (i.e. what are you building?), but I can offer a couple of suggestions.

First, you can scale a SurfaceView. If you look at the dumpsys output above, you'll notice that the "SurfaceView" line has a source crop rect of 320x240 (the size of the video), and a destination rect that's 984x738. This is coming from Grafika's "Play video (SurfaceView)", which sized the SurfaceView to preserve the 4:3 aspect ratio of the video. SurfaceFlinger takes care of scaling the content to match the view.

Second, if you're not showing DRM-protected video content, you can send it to a SurfaceTexture and just render it with GLES when you're rendering everything else. (This is exactly what TextureView does, which is why it requires hardware acceleration.) See e.g. "Continuous capture" in Grafika.

Update: a much longer description can be found in the Android System-Level Graphics doc.