1

I'm trying to play the same video at the same time in two different textureviews. I've used code from grafika (MoviePlayer and ContinuousCaptureActivity) to try to get it to work (thanks fadden). To make the problem simpler, I'm trying to do it with just one TextureView first.

At the moment I've created a TextureView, and once it get a SurfaceTexture, I create a WindowSurface and make it current. Then I generate a TextureID generated using a FullFrameRect object.

@Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int
        width, int height) {
    mSurfaceTexture = surface;
    mEGLCore = new EglCore(null, EglCore.FLAG_TRY_GLES3);
    Log.d("EglCore", "EGL core made");
    mDisplaySurface = new WindowSurface(mEGLCore, mSurfaceTexture);
    mDisplaySurface.makeCurrent();
    Log.d("DisplaySurface", "mDisplaySurface made");
    mFullFrameBlit = new FullFrameRect(new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
    mTextureID = mFullFrameBlit.createTextureObject();

    //mSurfaceTexture.attachToGLContext(mTextureID);
    clickPlayStop(null);
}

Then I get an off-screen SurfaceTexture, link it with the TextureID that I got above and create a surface to pass to a MoviePlayer thus:

public void clickPlayStop(@SuppressWarnings("unused") View unused) {
    if (mShowStopLabel) {
        Log.d(TAG, "stopping movie");
        stopPlayback();
        // Don't update the controls here -- let the task thread do it after the movie has
        // actually stopped.
        //mShowStopLabel = false;
        //updateControls();
    } else {
        if (mPlayTask != null) {
            Log.w(TAG, "movie already playing");
            return;
        }
        Log.d(TAG, "starting movie");
        SpeedControlCallback callback = new SpeedControlCallback();
        callback.setFixedPlaybackRate(24);

        MoviePlayer player = null;
        MovieTexture = new SurfaceTexture(mTextureID);
        MovieTexture.setOnFrameAvailableListener(this);
        Surface surface = new Surface(MovieTexture);
        try {
            player = new MoviePlayer(surface, callback, this);//TODO
        } catch (IOException ioe) {
            Log.e(TAG, "Unable to play movie", ioe);

            return;
        }

        adjustAspectRatio(player.getVideoWidth(), player.getVideoHeight());

        mPlayTask = new MoviePlayer.PlayTask(player, this);
        mPlayTask.setLoopMode(true);


        mShowStopLabel = true;
        mPlayTask.execute();
    }
}

The idea is that the SurfaceTexture gets a raw frame which I can use as an OES_external texture to sample from with OpenGL. Then I can call DrawFrame() from my EGLContext after setting my WindowSurface as current. 

private void drawFrame() {
    Log.d(TAG, "drawFrame");
    if (mEGLCore == null) {
        Log.d(TAG, "Skipping drawFrame after shutdown");
        return;
    }

    // Latch the next frame from the camera.
    mDisplaySurface.makeCurrent();
    MovieTexture.updateTexImage();
    MovieTexture.getTransformMatrix(mTransformMatrix);

    // Fill the WindowSurface with it.
    int viewWidth = mTextureView.getWidth();
    int viewHeight = mTextureView.getHeight();
    GLES20.glViewport(0, 0, viewWidth, viewHeight);
    mFullFrameBlit.drawFrame(mTextureID, mTransformMatrix);
    mDisplaySurface.swapBuffers();
}

If I wanted to do it with 2 TextureViews, the idea would be to call makeCurrent() and draw into each buffer for each view, then call swapBuffers() after the drawing is done.

This is what I want to do, but I am pretty sure this is not what my code is actually doing. Could somebody help me understand what I need to change to make it work?

@Fadden

Update: This is interesting. I changed the code in onSurfaceTextureAvailable to this:

@Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int
        width, int height) {
    mSurfaceTexture = surface;
    TextureHeight = height;
    TextureWidth = width;
    //mEGLCore = new EglCore(null, EglCore.FLAG_TRY_GLES3);
    Log.d("EglCore", "EGL core made");
    //mDisplaySurface = new WindowSurface(mEGLCore, mSurfaceTexture);
    //mDisplaySurface.makeCurrent();
    Log.d("DisplaySurface", "mDisplaySurface made");
    //mFullFrameBlit = new FullFrameRect(new Texture2dProgram(Texture2dProgram.ProgramType.OPENGL_TEST));
    //mTextureID = mFullFrameBlit.createTextureObject();
    //clickPlayStop(null);


    // Fill the SurfaceView with it.
    //int viewWidth = width;
    //int viewHeight = height;
    //GLES20.glViewport(0, 0, viewWidth, viewHeight);
    //mFullFrameBlit.drawFrame(mTextureID, mTransformMatrix);
    //mFullFrameBlit.openGLTest();
    //mFullFrameBlit.testDraw(mDisplaySurface.getHeight(),mDisplaySurface.getWidth());
    //mDisplaySurface.swapBuffers();
}

So, it shouldn't call anything else, just show the empty TextureView - and this is what I see...

Empty TextureView

Kongo
  • 71
  • 1
  • 2
  • 12
  • Your plan sounds fine. Use a single EGL context for both surfaces, then makeCurrent() + draw + swapBuffers() in each TextureView. Depending on your layout, you could also use a single TextureView and modify the GLES rendering to draw each on half of the View. – fadden Jul 15 '15 at 05:19
  • Cool. Thats good news. For the SurfaceTexture owned by the TextureView, i've added an "onFrameAvailable" listener. However, I keep getting an IllegalStateException: Unable to update texture contents when I call surfaceTexture.updateTexImage(); Is there some setup I need to do? – Kongo Jul 16 '15 at 11:10
  • From the SurfaceTexture documentation: "SurfaceTexture objects may be created on any thread. updateTexImage() may only be called on the thread with the OpenGL ES context that contains the texture object. The frame-available callback is called on an arbitrary thread, so unless special care is taken updateTexImage() should not be called directly from the callback." – fadden Jul 16 '15 at 15:34
  • Oh, so I need a Handler to call the update on the same thread?...Oh I see...thats why you have all these callbacks and handlers and things. Thanks – Kongo Jul 16 '15 at 22:55
  • The OpenGL ES API relies on thread-local storage, so you need to be very aware of what happens on which thread. It adds somewhat to the learning curve when working with this stuff. – fadden Jul 16 '15 at 23:51
  • There is some bizzare behaviour going on. I can tell that the movie is playing because the colours on the screen are changing, but the screen is uniform in colour. See edit – Kongo Aug 07 '15 at 23:10
  • Nothing seems blatantly wrong in the code. Usually YUV screw-ups tend to look more green (YUV 0,0,0 is medium green), which makes the solid blue a bit odd. My guess would be there's a mix-up in the textures somewhere, and either it's rendering from the wrong ID, or using the right ID but from the wrong context. Is everything happening on the UI thread? Are you able to render onto the TextureView with simple GL calls (e.g. draw a rect with glScissor / glClear like doAnimation() does in TextureViewGLActivity.java) from drawFrame()? – fadden Aug 08 '15 at 03:55
  • Actually, the beginning of the video is a blue screen. I think the exact same blue. Originally, I thought it may have been incorrect vectors because the colour does change as the video plays. Everything is happening on the UI thread I believe. The final surfacetexture, i just let do its work. I'll try some basic opengl stuff and write back – Kongo Aug 09 '15 at 04:16
  • So...check this out. No idea what this is....do you? – Kongo Aug 10 '15 at 12:55
  • I also tried drawing a simple triangle and something very bizarre is happening. The colours seem fine, I can change them no problems. The problem is in the vertex shader I think, but I am at a loss as to what exactly could be wrong. – Kongo Aug 10 '15 at 13:56
  • I've also used the 'show layout bounds' in the screenshot. The layout file i'm using has only the textureview inside a framelayout. But there seems to be extra stuff, and I don't know why the name of the activity is displayed so many times... – Kongo Aug 10 '15 at 14:02
  • Huh. The good news is, you can stop worrying about how all the video stuff fits together and just focus on GLES. Is this on a device or emulator? Which device / version of Android? Do you have other devices you can test on to compare to? The repeated Activity name is suspicious -- if you look closely it appears that the repeated bugdroid icons line up with the red stripes on the right edge, which makes me think it's caused by the same problem. – fadden Aug 10 '15 at 14:59
  • Haha. Well thats good. I was starting to feel confident that I understood how it all fit together, and then BAM. Confidence gone. This is on a Nexus 4, latest version of android (5.1.1). I don't have other devices, but I can try on an emulator. What could cause the extra bits in the layout if all i'm supposed to have is a textureview on the screen? – Kongo Aug 10 '15 at 22:20
  • I'm not really sure what's causing it. My recommendation is to start with something simple that works correctly, and gradually add bits and pieces until it falls over. (Or start from the broken code, and remove stuff until it stops acting weird.) – fadden Aug 11 '15 at 03:59
  • I've taken away pretty much everything except the TextureView. This is what it shows....is it normal? – Kongo Aug 14 '15 at 08:58
  • That image appears to have a duplicated nav bar, as if it's using the wrong texture. Do you see weird behavior in the Grafika demos (e.g. "simple GL in TextureView")? It's doing pretty much the same as you, though it's using a separate thread for rendering. Hmm. Are you rendering on it from the UI thread? If so, have you tried saving & restoring the previous EGL context when your rendering completes? (Wondering if the View UI is trying to render from the wrong EGL context.) – fadden Aug 14 '15 at 16:00
  • From that image, I don't even set up the EGL context, its just the TextureView by itself. I'm not rendering anything onto it. When I try to decode the video directly onto the surfaceTexture, the video is displaced downwards and I get a funny video-static looking thing on top where it seems to be bugging out. This has never happened before...only with this activity, other demos work fine – Kongo Aug 15 '15 at 00:33
  • I'll try rendering from another thread – Kongo Aug 15 '15 at 02:59

1 Answers1

2

Thanks to Fadden for the help.

So there seemed to be some unknown issue that was resolved when I used a new thread to decode and produce the frames. I haven't found out what caused the original problem, but I have found a way around it.

Kongo
  • 71
  • 1
  • 2
  • 12