1

Recently I am working with Google ML Kit Face Detection library. I implemented it successfully and able to detect my face through that. Played with contour points to draw a small bounding box over forehead.

During face detection it provide the image frames from Image Proxy. No my issue here is I'm getting only 6-7 frames per second. I want to increase the frame count approx. 25-30 frames per second.

I tried to decrease the target resolution from 640x480 to half of it, then I face 2 problems:

  1. Input Image quality is very bad.
  2. Due to which output image looses its YUV value and provide greyish green image.

Please help me with this to increase frame count per second.

Below are the library and code which I am using to achieve this task.

implementation 'com.google.mlkit:face-detection:16.1.5'
implementation 'androidx.camera:camera-core:1.0.2'
implementation 'androidx.camera:camera-camera2:1.0.2'
implementation 'androidx.camera:camera-lifecycle:1.0.2'
implementation 'androidx.camera:camera-view:1.0.0-alpha32'

private fun getImageBitmap(mediaImage: Image): Bitmap {
    val yBuffer = mediaImage.planes[0].buffer // Y
    val vuBuffer = mediaImage.planes[2].buffer // VU

    val ySize = yBuffer.remaining()
    val vuSize = vuBuffer.remaining()

    val nv21 = ByteArray(ySize + vuSize)

    yBuffer.get(nv21, 0, ySize)
    vuBuffer.get(nv21, ySize, vuSize)

    val yuvImage = YuvImage(nv21, ImageFormat.NV21, mediaImage.width, mediaImage.height, null)
    val outputStream = ByteArrayOutputStream()
    //yuvImage.compressToJpeg(cropRect, 100, outputStream)
    yuvImage.compressToJpeg(Rect(0, 0, yuvImage.width, yuvImage.height), 100, outputStream)
    val imageBytes = outputStream.toByteArray()
    val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
    val rotated = rotateBitmap(bitmap, 270F)
    val bos = ByteArrayOutputStream()
    rotated.compress(CompressFormat.JPEG, 100, bos) //100 is the best quality possibe
    val square = bos.toByteArray()
    return BitmapFactory.decodeByteArray(square, 0, square.size)
}

Please check the below image for reference

Check the frame count and input image size

markalex
  • 8,623
  • 2
  • 7
  • 32
Dnveeraj
  • 117
  • 3
  • 15

3 Answers3

0

Have you tried to set the option to PERFORMANCE_MODE_FAST? it says it will increase the return FPS, actually I have the same issue as you but I havn't got a chance to try it out.

Fred Guo
  • 31
  • 3
  • Fred Guo PERFORMANCE_MODE_FAST property is already set during Face Detection. And I'm getting 6-7 frames. – Dnveeraj May 14 '22 at 17:29
  • I have one question here: are you changing the camera capturing resolution from 640*480 to half of it, then any improvement for the FPS? In my understandings, there are three fps here: fps1 means the capturing fps from camera, fps2 means the output of MLkit(after face detector), the fps3 is that after you crop the rectangle of the forehead part. – Fred Guo Jun 02 '22 at 07:51
0

You are doing unnecessary format conversion in your code. Encoding the YUV image to JPEG then decoding back to RGB is going to take a toll on your performance.

CameraX provides a feature that integrates MLKit with CameraX. Please see MLKitAnalyzer. It also takes care of difficult tasks like coordinate transformation. You can take a look at the sample here: https://github.com/androidx/androidx/blob/androidx-main/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/MlKitFragment.kt

If you want to do it yourself, then I would suggest you to take a look at the source code of MLKitAnalyzer and do the same thing.

Xi 张熹
  • 10,492
  • 18
  • 58
  • 86
0

If you only want to detect faces within 2 meters from the camera(selfie usecase). You can try new Face mesh detection with BOUNDING_BOX_ONLY mode.

It is faster than old Face detection(fast mode):

"~14 ms on Pixel 3" vs "~60ms on Pixel 3"

jack
  • 125
  • 1
  • 9