0

I'm trying to capture Android's views as bitmaps and save them as .mp4 file.

I'm using MediaCodec to encode bitmaps and MediaMuxer to mux them into .mp4.

Using YUV420p color format I expect input buffers from MediaCodec to be of size resWidth * resHeight * 1.5 but Qualcomm's OMX.qcom.video.encoder.avc gives me more than that (no matter what resolution I choose). I believe that it wants me to do some alignment in my input byte stream but I have no idea how to find out what exactly it expects me to do.

This is what I get when I pack my data tightly in input buffers on Nexus 7 (2013) using Qualcomm's codec: https://www.youtube.com/watch?v=JqJD5R8DiC8

And this video is made by the very same app ran on Nexus 10 (codec OMX.Exynos.AVC.Encoder): https://www.youtube.com/watch?v=90RDXAibAZI

So it looks like luma plane is alright in faulty video but what happened with chroma plane is a mystery for me.

I prepared minimal (2 classes) working code example exposing this issue: https://github.com/eeprojects/MediaCodecExample

You can get videos shown above just by running this app (there will be same artefacts if your device utilizes Qualcomm's codec).

Android developer
  • 1,272
  • 1
  • 12
  • 17

1 Answers1

1

There are multiple ways of storing YUV 420 in buffers; you need to check the individual pixel format you chose. MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar and MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar are in practice the same, called planar or I420 for short, while the others, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar and MediaCodecInfo.CodecCapabilities.COLOR_TI_FormatYUV420PackedSemiPlanar are called semiplanar or NV12.

In semiplanar, you don't have to separate planes for U and V, but you have one single plane with pairs of interleaved U,V.

See https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java (lines 925-949) for an example on how to fill in the buffer for the semiplanar formats.

mstorsjo
  • 12,983
  • 2
  • 39
  • 62
  • Thank you for your help once again. What if codec supports `CodecCapabilities.COLOR_FormatYUV420Flexible` though? Documentation says that it can be both planar or semi-planar. Also would you be able to point some other traps I can fall into while trying to provide support for as many codecs as possible? – Android developer Oct 11 '16 at 07:35
  • 1
    If the codec supports `CodecCapabilities.COLOR_FormatYUV420Flexible` and you choose it, it allows the encoder to internally pick whichever format it wants (as long as it can be described as a flexible YUV420 format). If you use this, you need to use `MediaCodec.getInputImage` to get a description of the buffer (instead of the old `getInputBuffers()` or `getInputBuffer()`). Then you have three direct `ByteBuffers` for each plane, and a generic row and pixel stride to describe how to fill it. – mstorsjo Oct 11 '16 at 08:11
  • 1
    Even though the flexible YUV 420 format does allow really crazy formats, I think it will in practice actually be planar and semiplanar anyway, and those will still be exposed via the normal pixel formats. So if you care about supporting older versions, you don't really lose much by doing it the way you do (I still haven't seen a device that doesn't support either planar or semiplanar input, since this is what the CTS test requires as well!). – mstorsjo Oct 11 '16 at 08:13
  • 1
    So the main benefit of using the flexible YUV format is that it abstracts away these differences - if you have a content producer that can work with the `Image` class, this is probably simpler. It can also theoretically work better for filling buffers with uncommon sizes (e.g. not a multiple of 16) although I'm not sure if that actually is hooked up in practice - there's no CTS test for it as far as I know though. – mstorsjo Oct 11 '16 at 08:16