10

I'm working on a video encoding application which I want to prevent from stopping when the hosting Activity enters the background, or the screen cycles off/on.

The architecture of my encoder is derived from the excellent CameraToMpegTest example, with the addition of displaying camera frames to a GLSurfaceView (see Github links below). I'm currently performing background recording with a two-state solution:

  • When the hosting Activity is in the foreground, encode one video frame on each call to the GLSurfaceView.Renderer's onDrawFrame. This allows me access to the GLSurfaceView's EGL state in bursts so as not to block other events queued to the renderer thread.

  • When the hosting Activity enters the background, halt the onDrawFrame encoding and encode frames on another background thread within a loop. This mode is identical to the CameraToMpegTest example.

However if the screen is powered off the GLSurfaceView's EGLContext is lost and a new call to onSurfaceCreated occurs. In this case we have to re-create the EGL window surface connected to MediaCodec's input Surface. Unfortunately this 2nd call to eglCreateWindowSurface produces:

E/libEGL(18839): EGLNativeWindowType 0x7a931098 already connected to another API

Prior to calling, I release all EGL resources connected to the Android Surface.

Is there a way to swap the EGLSurface connected to MediaCodec's input Surface?

The complete source of my test application is on Github. Main Activity.

Update I applied the lessons learned here into a video sdk for Android based on the MediaCodec & MediaMuxer classes. Hope it helps!

dbro
  • 1,718
  • 1
  • 20
  • 34
  • 1
    `MediaCodec` shouldn't be affected by (or even aware of) being in the background. See e.g. the `screenrecord` command added in Android 4.4, which happily runs behind the scenes. The fact that it's encoding *anything* means it's still receiving input data, somy guess would be that something is affecting `Camera`. I don't understand why it would cause the preview `Surface` to be blank while the preview `byte[]` has real data. – fadden Nov 27 '13 at 17:00
  • Updated my question. I'm now able to transition to background recording (no GLSurfaceView display) and then foreground recording (GLSurfaceView display) **except** when a screen off/on event occurs in the interim... – dbro Dec 03 '13 at 22:11
  • You're missing a `glSurfaceView.onPause()` in your main activity onPause(). Not sure if that will matter though. I may be able to play with it a bit tomorrow and see if I can replicate the behavior. – fadden Dec 04 '13 at 06:04
  • The Activity's `onPause()` calls a method that uses `glSurfaceView.queueEvent()` to queue an operation that must occur on the renderer thread before `glSurfaceView.onPause()` pauses it. [See here](https://github.com/OnlyInAmerica/FFmpegTest/blob/service/src/com/example/ffmpegtest/recorder/HLSRecorder.java#L226) – dbro Dec 04 '13 at 17:47
  • Screen off/on causes the GLSurfaceView's EGLContext to be lost, so we've got to create a new CodecInputSurface.mEGLSurface that shares the new EGLContext. However having trouble disconnecting the MediaCodec input Surface to avoid `already connected to another API` error on `eglCreateWindowSurface`. Should this be possible? – dbro Dec 12 '13 at 19:05
  • Quick thought: http://developer.android.com/reference/android/opengl/GLSurfaceView.html#setPreserveEGLContextOnPause(boolean) – fadden Dec 12 '13 at 20:51
  • I have `setPreserveEGLContextOnPause` set `true`. Works until the screen off configuration change causes onDestroy(). Want to release this as a library, so don't want to require developers handle configuration changes manually. At worst I can swap in a new MediaCodec and update the Surface. – dbro Dec 12 '13 at 21:12
  • Looking at `~egl_surface_t` in `frameworks/native/opengl/libs/EGL/egl_object.cpp`, the previous EGL surface *should* disconnect from the MediaCodec input surface when it's destroyed (eglDestroySurface + eglMakeCurrent(nothing)). One thing you might do is add checks to `CodecInputSurface` methods... call `eglGetCurrentContext` and see if it's `.equal()` to `mEGLEncodeContext` to confirm that the right context is current when you destroy+eglMakeCurrent (don't use "=="). The Surface won't be destroyed if it's still current in an orphaned context. – fadden Dec 12 '13 at 21:44
  • What I should have said is: make sure you're in the right *thread* when you call `eglMakeCurrent(nothing)`. If the context+thread is current in thread #1, then calling `eglMakeCurrent(nothing)` in thread #2 won't release the thread and surface. If the thread's current context matches what you have in the object, then you know you're in the right place. (Unless you made the same context current in two different threads, in which case you're going to be in a world of hurt.) – fadden Dec 12 '13 at 23:03
  • FWIW, I have something similar working across power-cycle on a Nexus 5. My encoder thread gets a message with the new GLSurfaceView EGLSurface, and calls the equivalent of releaseAllButSurface() + eglSetup() + SurfaceTextureManager.surfaceCreated(). My app just goes to sleep while the power is off, so you end up with a long pause in the recording, but it picks right up when the app resumes. – fadden Dec 12 '13 at 23:22

1 Answers1

10

Background first...

When you call eglCreateWindowSurface(), the Android EGL wrapper calls native_window_api_connect() on the Surface you passed in. This eventually turns into a BufferQueue producer connect call, which means that this EGL surface is now the sole source of graphics buffers for the Surface.

The EGL surface stays connected to the Surface until the EGL surface is destroyed. When it is, the surface destructor calls native_window_api_disconnect() to disconnect the EGL surface from the BufferQueue. The EGL surface is reference-counted, with the refcount incremented when the surface is passed to eglMakeCurrent() so to be destroyed two things must happen:

  1. eglDestroySurface() must be called
  2. the EGL surface must not be "current" in any thread

The second item requires calling eglMakeCurrent() with another EGL surface (or EGL_NO_SURFACE), or calling eglReleaseThread(), on any thread that had previously used the surface. One quick way to confirm that this is done is to add logging before calls to eglMakeCurrent() when the surface is made current and un-current, and compare the thread IDs by viewing the logcat output with adb logcat -v threadtime. It may also be useful to use EGL queries like eglGetCurrentSurface(EGL_DRAW) to confirm that you're doing the un-current in the thread that has made the surface current.

If the EGL surface isn't destroyed, it won't disconnect from the Surface, and attempts to connect a new producer (by calling eglCreateWindowSurface with a new EGL surface) will be rejected with the "already connected" message.

Update: My implementation is now available in the Grafika test project. If you install this, select "Show + capture camera", start recording, toggle the power, and then stop recording, you should have a complete movie with a long pause in the middle. You can back out, select "Play video", and choose "camera-test.mp4" to view it.

fadden
  • 51,356
  • 5
  • 116
  • 166
  • 1
    Eureka! Grafika is beautiful. Seeing all the moving pieces properly modularized makes my craft smell of crayons and glue. – dbro Dec 21 '13 at 00:41
  • @fadden In ContinuousCaptureActivity.java(https://github.com/google/grafika/blob/master/src/com/android/grafika/ContinuousCaptureActivity.java), When onPause, you release mDisplaySurface first, then release mEglCore. The mDisplaySurface.release() will at last call EGL14.eglDestroySurface, but at that moment you did not confirm the "current" issue. Will the eglDestroySurface succeed? – dragonfly Sep 18 '15 at 10:11
  • I'm not sure what you mean by "did not confirm the 'current' issue". As noted in the answer, the resources are reference-counted; being current adds an additional reference. So the surface is discarded when `eglDestroySurface()` is called *and* it is no longer current. There *is* a bug related to this code -- https://github.com/google/grafika/issues/24 -- but that's more about what I failed to do in `onResume()`. – fadden Sep 18 '15 at 15:04