Is it possible/how to feed MediaCodec decoded frames to MediaCodec encoder directly?

Question

My goal is to splice video fragments from several video files. Fragments are defined by the arbitrary start time and end times. Initially I wanted to do it using a library like mp4parser but it can only cut streams at sync (IFRAME) points, while I need higher precision.

My scenario is Extract encoded stream from file -> Decode -> Encode -> Mux the result into mp4 file. Right now generally code works but resulted video is white noise. Tested on Nexus-S and Galaxy-S3. My code is a combination of several examples:

• Read previously recorded files based on MoviePlayer.java
• Decode-Encode: DecodeEditEncodeTest.java
• Mux video stream into mp4 - yet another example, not relevant here

I want to simplify the examples because I do not need to process frames in the middle. I've tried to feed buffers from decoder output to encoder input without Surface in the middle. The overall process worked in the sense that code run to completion and resulted in playable video file. However the contents of the file is white noise.

That's a snippet of code that feeds the frames from decoder to encoder. What is wrong and how to make it work?

...
} else { // decoderStatus >= 0
    if (VERBOSE) Log.d(TAG, "surface decoder given buffer "
                            + decoderStatus + " (size=" + info.size + ")");
    // The ByteBuffers are null references, but we still get a nonzero
    // size for the decoded data.
    boolean doRender = (info.size != 0);
    // As soon as we call releaseOutputBuffer, the buffer will be forwarded
    // to SurfaceTexture to convert to a texture.  The API doesn't
    // guarantee that the texture will be available before the call
    // returns, so we need to wait for the onFrameAvailable callback to
    // fire.  If we don't wait, we risk rendering from the previous frame.
    //   decoder.releaseOutputBuffer(decoderStatus, doRender);
    if (doRender) {
    // This waits for the image and renders it after it arrives.
//                  if (VERBOSE) Log.d(TAG, "awaiting frame");
//                          outputSurface.awaitNewImage();
//                          outputSurface.drawImage();
//                          // Send it to the encoder.
//                              inputSurface.setPresentationTime(info.presentationTimeUs * 1000);
//                          if (VERBOSE) Log.d(TAG, "swapBuffers");
//                          inputSurface.swapBuffers();

            encoderStatus = encoder.dequeueInputBuffer(-1);

            if (encoderStatus >= 0) {
                                encoderInputBuffers[encoderStatus].clear();

                                decoderOutputBuffers[decoderStatus].position(info.offset);
                                decoderOutputBuffers[decoderStatus].limit(info.offset + info.size);

                                encoderInputBuffers[encoderStatus].put(decoderOutputBuffers[decoderStatus]);
                                encoder.queueInputBuffer(encoderStatus, 0, info.size, info.presentationTimeUs*1000, 0);
                }
            }

                        decoder.releaseOutputBuffer(decoderStatus, false);
...

Answer 1

It's much better to use a Surface than a ByteBuffer. It's faster as well as more portable. Surfaces are queues of buffers, not just framebuffers for pixel data; decoded video frames are passed around by handle. If you use ByteBuffers, the video data has to be copied a couple of times, which will slow you down.

Create the MediaCodec encoder, get the input surface, and pass that to the decoder as its output surface.

If you need to work with API 16/17, you're stuck with ByteBuffers. If you search around you can find reverse-engineered converters for the wacky qualcomm formats, but bear in mind that there were no CTS tests until API 18, so there are no guarantees.