3

So I am trying to decode a stream of raw h264 data and render it to a surface on Android. Here are the steps:

  1. Receive a packet of h264 stream
  2. Accumulate it and try to extract NAL units (byte sequences starting with 00 00 00 01 (NAL header) and up until the next NAL header.
  3. For every extracted NAL unit, call feedFrame(data) where data is a byte[] that starts with NAL header and contains the extracted unit.
  4. See the video rendered on the surface I provided.

The following code does utilizes the AVC decoder:

public StreamReceiver(DashCamActivity activity, Surface surface, int width, int height, byte[] sps, byte[] pps) {
    this.activity = activity;
    decoder = MediaCodec.createDecoderByType("video/avc");

    format.setByteBuffer("csd-0", ByteBuffer.wrap(sps));
    format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
    decoder.configure(format, surface, null, 0);
    decoder.start();

}

public void shutdown() 
{
    decoder.stop();
    decoder.release();
}

public void feedFrame(byte[] data) 
{
    BufferInfo info = new BufferInfo();
    int inputIndex = decoder.dequeueInputBuffer(1000);
    if(inputIndex == -1)
        return;
    ByteBuffer inputBuffer = decoder.getInputBuffers()[inputIndex];
    if (inputIndex >= 0) {
        inputBuffer.clear();
        inputBuffer.put(data, 0, data.length);
        inputBuffer.clear();
        decoder.queueInputBuffer(inputIndex, 0, data.length, 0, 0);
    }

    int outIndex = decoder.dequeueOutputBuffer(info, 1000);

    switch (outIndex) {
    case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
        break;

    case MediaCodec.INFO_TRY_AGAIN_LATER:
        break;

    case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
        break;

    default:
        decoder.releaseOutputBuffer(outIndex, true);
        break;
    }
}

For smaller resolutions (1024x768, 1280x800) everything works perfectly. However with larger resolutions (1920x1080, 1900x600), where the length of the byte array I provide is above 65535 (64k), the video starts having stutters and artifacts and Logcat reports strange decoder errors (e.g. IOCTL_MFC_DEC_EXE failed(ret : -2001) on Galaxy S3). This also happens on a relatively new device that can play 4k with twice the framerate I provide. So I must be doing something wrong, and I don't know if my 64k theory has any truth in it, it's merely an observation.

So to recap:

  • I am providing individual NAL units to the decoder, starting with the header.
  • The h264 stream is of baseline profile, level 4.0.
  • Writing the contents of the NAL units to a file in the order they arrive produces a video file that is fully playable using the basic media players

How do I get it to play at high resolutions?

Vladimir Gazbarov
  • 860
  • 1
  • 10
  • 25
  • Well, you're only giving it 1000us to decode a frame. If the frame isn't ready by then, do you have sufficient logic in place to execute a re-try, or do frames just start queuing up in the decoder? – greeble31 Mar 25 '19 at 21:11
  • There is no retry logic since I don't know if a frame is to be decoded after a given moment. A frame consists of unknown number of NALs. I don't now where a new frame starts and I have no choice but to simply feed them and hope video comes out – Vladimir Gazbarov Apr 09 '19 at 22:01
  • Seems to me your choice of architecture has left you with a number of unhandled edge cases (that are exacerbated by higher resolutions). Why don't you switch the codec to asynchronous operation? Saves you from having to block for N microseconds when trying to dequeue an output buffer. You'll have to rethink how you queue input buffers, though. – greeble31 Apr 10 '19 at 18:35

0 Answers0