7

I'm working on a Android app that uses MediaCodec to encode H.264 video using the Surface method. I am targeting Android 5.0 and I've followed all the examples and samples from bigflake.com (I started working on this project two years ago, so I kind of went through all the gotchas and other issues).

All is working nice and well on Nexus 6 (which uses the Qualcomm hardware encoder for doing this), and I'm able to record flawlessly in real-time 1440p video with AAC audio, in a multitude of outputs (from MP4 local files, upto http streaming). But when I try to use the app on a Sony Android TV (running Android 5.1) which uses a Mediatek chipset, all hell breaks loose even from the encoding level. To be more specific:

  1. It's basically impossible to make the hardware encoder work properly (that is, "OMX.MTK.VIDEO.ENCODER.AVC"). With the most basic setup (which succeeds at MediaCodec's level), I will almsot never get output buffers out of it, only weird, spammy, logcat error messages stating that the driver has encountered errors each time a frame should be encoded, like this:

    01-20 05:04:30.575 1096-10598/? E/venc_omx_lib: VENC_DrvInit failed(-1)!
01-20 05:04:30.575 1096-10598/? E/MtkOmxVenc: [ERROR] cannot set param
01-20 05:04:30.575 1096-10598/? E/MtkOmxVenc: [ERROR] EncSettingH264Enc fail

  2. Sometimes, trying to configure it to encode at a 360 by 640 pixels resolution will succeed in making the encoder actually encode stuff, but the first problem I'll notice is that it will only create one keyframe, that is, the first video frame. After that, no more keyframes are ever, ever created, only P-frames. Ofcourse, the i-frame-interval was set to a decent value and is working with no issues on other devices. Needless to say, this makes it impossible to create seekable MP4 files, or any kind of streamable solution on top.

  3. Most of the times, after releasing the encoder, logcat will start spamming endlessly with "Waiting for input frame to be released..." which basically requires a reboot of the device, since nothing will work from that point on anyway.

  4. In the case where it doesn;t go havoc after a simple release(), no problem - the hardware encoder is making sure that it cannot be created a second time, and it falls back to the generic SOFTWARE avc google encoder. hich ofcourse is basically a mockup encoder which does little to nothing than spit out an error when trying to make it encode anything larger than 160p videos...

So, my question is: is there any hope of making this MediaCodec API actually work on such a device? My understanding was that there are some CTS tests performed by Google/manufacturers (in this case, Sony) that would allow a developer to actually think that an API is supported on a device which prouds itself as running Android 5.1. Am I missing something obvious here? Did anyone actually ever tried doing this (a simple MediaCodec video encoding test) and succeeded? It's really frustrating!

PS: it's worth mentioning that not even Sony provides yet a recording capability for this TV set, which many people are complaining anyway. So, my guess is that this sounds more like a Mediatek problem, but still, what exactly are the Android's CTS for in this case anyway?

Adrian Crețu
  • 878
  • 8
  • 17
  • Some of the code on bigflake.com is the actual 4.3-era CTS tests (you'll notice it's just linking to the git repository). I'd expect there to be even more tests for a 5.x system, and the device must pass those tests to be certified as Android. There are always going to be gaps in the tests though. One thing off the top of my head is that 360 does not divide evenly by 16, and some codecs have a secret "must be divisible by 16" requirement, though usually you just get bad frames. It might be worth trying to get Grafika to work on the TV and see how it fares (https://github.com/google/grafika). – fadden Jan 20 '16 at 06:18
  • Thanks! I ran Grafika on the TV and weirdly enough, the GL recording worked (at 12fps, many frames dropped). However, what I am doing is that I'm using the MediaProjection API to capture the screen, and it's projecting on the encoder's surface, which is not the same (OES texture, etc). It looks more and more like this is an OpenGL problem. For example, while I'm capturing the screen and try to use YouTube or Kodi to view a video, the screen is black (has audio), but the output surface of the virtual display is written with the correct decoded video which should have been rendered on screen. – Adrian Crețu Jan 20 '16 at 13:56
  • So, after more retries and investigation, my conclusion is that the Mediatek AVC encoding drivers are immature, and for every single input frame it adds 20 lines of logcat entries full of debug info and cryptic errors. The only working encoder resolution that works looks to be 1280 x 720 (probably because it's the only one multiple of 16), and that one works terrible too, with a FPS of 7-8 in the output. – Adrian Crețu Jan 28 '16 at 02:40
  • The CTS tests like EncodeDecodeTest exercise three resolutions: QCIF (176x144), QVGA (320x240), and 720p (1280x720). So those are the most likely to work. There is no performance check in the tests I worked on, but I would hope the Camera or MediaRecorder tests have something. It's possible this device is not expected to be used at 720p resolution. The MediaCodecInfo.CodecProfileLevel information would hopefully indicate that. – fadden Jan 28 '16 at 05:48
  • 1
    Thanks for your patience :) I iterated over the MediaCodecList and yes, the size-range is maxed out at 1280x720 (this being a 2015 4K TV...). However it doesn't explain rest of the problems: why there's only one key-frame generated, endless "waiting for release.." even after app is killed; black screen while projecting from virtual display to codec surface. Probably ffmpeg would at least, work, too bad HW encoder doesn't look to be ever supported natively for this device. I'm curious how the engineers at Sony will work around these problems (providing TV record functionality), if ever :) – Adrian Crețu Jan 30 '16 at 00:39

0 Answers0