4

I have an example application (full source) that encodes camera frames with MediaCodec while displaying them on a GLSurfaceView.

Systrace confirms 30 drawing calls are made each second:

Systrace screenshot

However, a screenrecord recording (.mp4, YouTube) shows the apparent framerate to be considerably lower.

In a nutshell, my encoding & display loop does the following:

  • Makes a MediaCodec Surface input EGL Context current
  • Draws a camera frame to the MediaCodec EGL surface
  • Makes the GLSurfaceView EGL Context current
  • Draws the same camera frame to the GLSurfaceView

On a Galaxy Nexus LTE and Nexus 7 (both with AOSP 4.4), the application performs as expected. So far only the Nexus 5 experiences this discrepancy between the number of frames drawn to the screen and the number of frames apparent...

I pray I'm not insane.

dbro
  • 1,718
  • 1
  • 20
  • 34
  • The `screenrecord` command adds considerable overhead -- surfaceflinger needs to composite all layers for a virtual display using GLES -- which can adversely affect your frame rate. Take a look at systrace while `screenrecord` is running. – fadden Dec 04 '13 at 23:41
  • I experience the issue without screenrecord as well. It's totally night and day between the N5 and the other devices I've tried. – dbro Dec 04 '13 at 23:54
  • The animated arrow in the top-left corner seems to be animating more often than the camera display, so it's not a composition issue. I've been trying to investigate some horrifying glReadPixels() performance in http://bigflake.com/mediacodec/#ExtractMpegFramesTest on the Nexus 5 (177ms for a single 720p frame, which goes down to 6ms if the video decoder isn't involved); it's possible this is related. – fadden Dec 05 '13 at 01:17
  • 1
    Similar problem (with possible workaround): http://stackoverflow.com/questions/20396515/surfacetexture-updateteximage-to-shared-2-eglcontexts-problems-on-android-4-4 – fadden Dec 05 '13 at 15:39
  • 1
    Thanks! I'm looking into that. FWIW I have the [systrace of my application on the N5](https://s3.amazonaws.com/dbro/n5_glsurfaceview/systrace_n5_ffmpegtest.html) (16MB. Chrome crashes when I load from that url, but not when I open it locally...). Also worth noting that all my test devices were on 4.4, and only the N5 produced this issue with my app. – dbro Dec 05 '13 at 19:24
  • 1
    Looking about 1.5s in... if you look at the SurfaceView line near the top you can see how many buffers are in its queue at any time. The buffer source isn't keeping the queue fed, so SurfaceFlinger isn't drawing anything -- on the surfaceflinger line you can see it composite, wait a vsync (17ms), composite, wait 3x vsync, and repeat. There's a lot of CPU going on but much of it seems to be PackageManager. Thread-445 shows 30-40ms between calls to awaitImage(), which is often taking 20ms to return. Something has definitely gummed up the works. – fadden Dec 05 '13 at 22:13
  • FYI re: earlier comment... ExtractMpegFramesTest slowdown turned out to be my fault -- glReadPixels() does poorly if you don't specify an alpha plane in your EGLConfig. So this problem is probably different. – fadden Dec 06 '13 at 02:04

1 Answers1

10

I was able to replicate the behavior, and my GL wizard office-mate figured out the problem.

Basically, one of the EGL contexts isn't noticing that the texture contents have changed, so it keeps rendering older data. We think it's getting occasional updates because it has a set of buffers that it cycles through, so eventually it re-uses the buffer you're looking at.

I was able to fix the problem in my code by updating the texture renderer class, changing this:

GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, mTextureID);

to this:

GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, mTextureID);

The un-bind and re-bind causes the driver to pick up the right buffer.

fadden
  • 51,356
  • 5
  • 116
  • 166