How to use MediaMuxer / MediaCodec InputSurface with GLSurfaceView?

Question

Following the BigFlake example there is a comment that states :

// Acquire a new frame of input, and render it to the Surface.  If we had a
// GLSurfaceView we could switch EGL contexts and call drawImage() a second
// time to render it on screen.  The texture can be shared between contexts by
// passing the GLSurfaceView's EGLContext as eglCreateContext()'s share_context
// argument.

I use EGL14.getCurrentContext() to query the current context and pass it into the EGL14.eglCreateContext() share_context parameter, but How do you "switch EGL contexts"?

The GLSurfaceView and MediaCodec.inputSurface have two different surfaces and two different contexts, so I assume you just call eglMakeCurrent() on each set individually, is that correct? Do you need to eglDestroyContext() or eglDestroySurface()?

Added update

Thanks Fadden, I think I figured out this error, instead of drawFrame I was calling drawImage, but you shouldn't have to update the image a second time, right?

Now I am getting an out of memory error glError 1285 when dequeueBuffer is called after EOS has been set??? Maybe I am calling it after the recording has stopped. Thanks for your help.

create EGLSurface in MyEGLWrapper.java

saveToScreenRenderState();
mEGLContext = EGL14.eglCreateContext(mEGLDisplay, configs[0], mScreenEglContext, 
        contextAttribs, 0);
checkEglError("eglCreateContext");

// Create a window surface, and attach it to the Surface we received.
mEGLSurface = EGL14.eglCreateWindowSurface(mEGLDisplay, configs[0], mSurface,
        surfaceAttribs, 0);
checkEglError("eglCreateWindowSurface");

...

private void saveToScreenRenderState() {
    //System.arraycopy(mProjectionMatrix, 0, mSavedMatrix, 0, mProjectionMatrix.length);
    mScreenEglDisplay = EGL14.eglGetCurrentDisplay();
    mScreenEglDrawSurface = EGL14.eglGetCurrentSurface(EGL14.EGL_DRAW);
    mScreenEglReadSurface = EGL14.eglGetCurrentSurface(EGL14.EGL_READ);
    mScreenEglContext = EGL14.eglGetCurrentContext();
}

public void makeCurrent(boolean toScreen, Surface surface) {
    if (toScreen) { //as opposed to toEncoder
        makeScreenSurfaceCurrent();
        return;
    } 
    EGL14.eglMakeCurrent(mEGLDisplay, mEGLSurface, mEGLSurface, mEGLContext);
    checkEglError("eglMakeCurrent");
}

private void makeScreenSurfaceCurrent() {
    EGL14.eglMakeCurrent(mScreenEglDisplay, mScreenEglDrawSurface, 
            mScreenEglReadSurface, mScreenEglContext);
    checkEglError("eglMakeCurrent");
}

In CaptureManager.java

private void drawFrameOnInputSurface() {
    //draw a second time for inputSurface 
    mEGLWrapper.makeCurrent(false, videoCodec.videoCodecInputSurface);
    drawFrame(false);
    //ByteBuffer frame = mStManager.drawImage();
    videoCodec.runQue(false); // clear que before posting should this be on this thread???
    mEGLWrapper.setPresentationTime(mSt.getTimestamp(),!recordingStopped);
    mEGLWrapper.swapBuffers(!recordingStopped);
    videoHandler.post(new Runnable(){
        @Override
        public void run(){
            videoCodec.updateFrame(!isRecording);
        }
    });
}

private void drawFrame(boolean updateImage){
    mStManager.drawImage(updateImage);
    mGLView.mRenderer.drawFrame();
}

In SurfaceTextureManager.java

public ByteBuffer drawImage(boolean updateImage) {
    // Latch the data.
    if (updateImage){
        mTextureRender.checkGlError("before updateTexImage");
        mSurfaceTexture.updateTexImage();
    }
    return mTextureRender.drawFrame(mSurfaceTexture);
}

error in SurfaceTextureManager.java on mSurfaceTexture.updateTexImage();

FWIW, I got this working in a pretty [simple, straightforward project](https://github.com/OnlyInAmerica/FFmpegTest/tree/gldisplay). One activity hosting the GLSurfaceView, and one class managing all the MediaCodec calls. Happy to chat about it. — dbro, Nov 26 '13 at 17:26
Thanks, I got this working. needs a little help lining up the audio for longer clips, but records playable mp4s for shorter clips just find. I built your code a few weeks back. Good example. — HPP, Dec 03 '13 at 20:28

score 5 · Accepted Answer · answered Oct 31 '13 at 14:55

5

Yes, use eglMakeCurrent to switch between the two contexts.

You do not need to destroy the contexts or surfaces you're not currently using (until you're shutting things down, anyway).

For an example of rendering twice with a shared context, see the bigflake Breakout game recorder patch.

answered Oct 31 '13 at 14:55

fadden

51,356
5
116
166

Thanks so much, I should have seen that example. I read your first two examples and got have been lost in my implementation for the past month, but it is starting to come together. It is actually not very complicated now, I would recommend anyone working on this stuff to read up on [EGL](http://www.khronos.org/registry/egl/) first. It will save a lot of time when everything makes sense. – HPP Nov 07 '13 at 00:13
It didn't work, In contradiction to this answer your answer to [dbro's question](http://stackoverflow.com/questions/19102308/mediacodec-with-surface-input-producing-chunked-output) states that you need to destroy and recreate the surface ??? I should clarify I want the camera preview to be displayed onscreen. What is the easiest or lowest cost method to do this? Is there a way to copy the `textureSurface` to a texture owned by another context? – HPP Nov 07 '13 at 19:38
1

dbro's question was about MediaCodec input surfaces. You can't "restart" an encoder, and the input Surface is tied to the encoder, so you have to create a new MediaCodec + input Surface pair for each segment of video. Note that a `Surface` and an `EGLSurface` are not the same thing -- the former is an Android concept, the latter is an OpenGL concept, and the two are totally unconnected until you call `eglCreateWindowSurface()`. – fadden Nov 07 '13 at 21:06
I'm not sure what you mean by `textureSurface`. Using `SurfaceTexture` to catch the Camera preview, the frame is available as an "external texture" that can be used as the image source. You need two EGL contexts, one for the video output, one for `GLSurfaceView`; you want to create the former while the latter is current, and pass `eglGetCurrentContext()` to `eglCreateContext()` as shown in the Breakout example. Then you just render twice, changing contexts, as shown in `GameSurfaceRenderer#onDrawFrame()`. Do all of this on the same thread. Watch logcat for errors. – fadden Nov 07 '13 at 21:15
I'm sorry let me rephrase, I'm getting the same error dbro got, `checkAndUpdateEglState: invalid current EGLContext` in `mSurfaceTexture.updateTexImage();` The code is [here](https://github.com/hpp/penelopefree/blob/master/src/com/harmonicprocesses/penelopefree/camera/CaptureManager.java) but it is kind of a spaghetti mess. Let me update the question with the relevant code. – HPP Nov 08 '13 at 00:12
The current `EGLDisplay` and `EGLContext` must be the same every time you call `updateTexImage()`, or you will get that message from `GLConsumer.cpp` (see dbro answer for source link). You can switch away and do work on a different context afterward, but you have to switch back before calling `updateTexImage()` again. The display and context are latched the first time `updateTexImage()` executes. – fadden Nov 08 '13 at 00:21

How to use MediaMuxer / MediaCodec InputSurface with GLSurfaceView?

1 Answers1