1

I'm building an internal shared library for Android platform. I have the signing keystore from the device manufacturer.

My library is making use of ScreenRecord.cpp internal file from Android source. Recording works fine with the MediaCodec encoder; however I want to access each frame so that I can apply some image overlay logo on to each frame before it gets passed to the encoder. There's an overlay example in Android source too, but that only works for newer versions of Android (5.0 / API 21+). I want to have an overlay solution for Android Kitkat (4.4 / API 19)

Here's a code example that I obtained from minicap.

mVirtualDisplay = android::SurfaceComposerClient::createDisplay(
android::String8("minicap"),
true);

LOGI("Creating buffer queue");
mScreenshotClient.getCpuConsumer();
mBufferQueue = mScreenshotClient.mBufferQueue;

LOGI("Creating CPU consumer");
mConsumer = new android::CpuConsumer(mBufferQueue, 3, false);
mConsumer->setName(android::String8("minicap"));
mConsumer->setDefaultBufferSize(targetWidth, targetHeight);
mConsumer->setDefaultBufferFormat(android::PIXEL_FORMAT_RGBA_8888);

mConsumer->setFrameAvailableListener(mFrameProxy); 
//mFrameProxy is from:
//class FrameProxy: public android::ConsumerBase::FrameAvailableListener

LOGI("Publishing virtual display");
android::SurfaceComposerClient::openGlobalTransaction();
android::SurfaceComposerClient::setDisplaySurface(mVirtualDisplay, mBufferQueue);
android::SurfaceComposerClient::setDisplayProjection(mVirtualDisplay,
android::DISPLAY_ORIENTATION_0, layerStackRect, visibleRect);
android::SurfaceComposerClient::setDisplayLayerStack(mVirtualDisplay, 0);// default stack

android::SurfaceComposerClient::closeGlobalTransaction();

I set up the above code, but onFrameAvailable() method of FrameAvailableListener gets called only once. It never gets called again even when I do stuff on the screen. What am I missing here? Isn't there any lesser tricky way to access the frames before passing to the encoder?

Usman.3D
  • 1,791
  • 3
  • 16
  • 27

1 Answers1

2

An example of adding an overlay is built into the screenrecord sources for Lollipop. As far as I can recall it doesn't rely on any features added in Lollipop, so you should be able to build and run it on 4.4. As noted on bigflake, the --bugreport mode was added to AOSP in the 4.4 time frame, but didn't actually ship with the system until 5.x. (With a minor tweak, it should even run on 4.3, but I haven't tried it.)

The key source files are Overlay.{cpp,h}. It does the same thing that you would do from code written in Java: create a GLConsumer (SurfaceTexture), use that to convert the incoming frames to GLES textures, then render texture + overlay to the video encoder.

Sample video is here. Note it adds a block of text to the very start, and a running timestamp / frame-counter to the top left corner.

Note for anyone else reading this: this code is using internal private APIs that have been changing in recent releases, so any binaries must be built for specific versions of Android, and may not be portable to devices built by different manufacturers even if they run the same version of Android (sometimes the OEMs like to mess with things).

Update: My earlier statements about working on KitKat weren't accurate -- there was a major API shift before the Lollipop version went out. The trick is to grab the sources before this change went in, as that was when the BufferQueue API rewrite reached screenrecord. You can see from the change list that the --bugreport option went in about five months before that.

fadden
  • 51,356
  • 5
  • 116
  • 166
  • Thanks but I tried to make use of that code but it kept asking for dependencies that were only to be found in Lollipop. Specifically: CreateBufferQueue function does not exist in earlier versions of Android, and I'm not sure how to replicate that functionality in older sources. I did try to replicate but then CreateBufferQueue has dependency on BufferQueueCore that is not found in Android Kitkat. – Usman.3D Sep 17 '15 at 15:13
  • Despite, Android's source is so complicated and I don't find any elaborate class diagrams explaining things in intuitive way. For example: setDisplaySurface should accept IGraphicBufferConsumer, but BufferQueue instance is being passed. Now BufferQueue is not child of IGraphicBufferConsumer. It's confusing at times. – Usman.3D Sep 17 '15 at 15:17
  • Your breakout example is great but how does it suit my needs? I have to grab the screen first, the only way I saw was through Virtual Display (correct me if I'm wrong), that's because I don't have control over drawing the contents on to the surface (unlike breakout where we draw our objects). My library should grab the screen frames, draw the logo and put it to the encoder. Is that all possible through breakout example? – Usman.3D Sep 17 '15 at 15:26
  • 1
    The Breakout patch (or, better, "record GL app" in Grafika) works for a single app when all you care about is the GLES surface. If you want the whole screen, including View UI, nav bar, notifications, etc., you need to use a virtual display. Ditto if you're not in control of the GLES rendering. I was wrong about the usefulness of the 'L' release code, but if you roll it back to the change before Dan Stoza's first checkin it should be much better (he re-wrote the BufferQueue API) -- see notes added to answer. – fadden Sep 17 '15 at 15:35
  • Awesome. I just checked the Overlay's history and found Dan Stoza's commit. I'll checkout the code prior to his commit and test it. Will get back to you later on. Thanks – Usman.3D Sep 17 '15 at 15:45
  • 1
    Other notes: CPUConsumer is for CPU access to the YUV frame data, e.g. for encoding with libjpeg. GLConsumer is for GLES "external" texture access to the frame. Since you're adding an overlay you'd want GLConsumer. IGraphicBufferConsumer is the Binder IPC interface to the consumer side of the BufferQueue. There's some C preprocessor Binder-declaration trickery that goes on with IInterface <-> BnInterface that can make it hard to follow names around in the code; in the KitKat sources you can see that BufferQueue inherits from BnGraphicBufferConsumer. – fadden Sep 17 '15 at 16:05