0

I am trying to use MLKit in my app for face detection. I am using Camera 2 API to get the image from image reader and pass it to the face detector for processing. This works very well on devices like Motorola, Readmi, Realme and, also most of Samsung devices but fails on Asus. I checked the log and found out that

setOnImageAvailableListener

is called only once instead of calling for every frame.

Attaching my code snippet below.

imageReader = ImageReader.newInstance(
           480,
            640
            ImageFormat.YUV_420_888,
            2
        )

val previewSurface = Surface(texture)
        mPreviewBuilder!!.addTarget(previewSurface)

val recordingSurface = imageReader?.surface!!
mPreviewBuilder?.addTarget(recordingSurface!!)

mCameraDevice!!.createCaptureSession(
            listOf(previewSurface, recordingSurface),
            object : CameraCaptureSession.StateCallback() {
                override fun onConfigured(@NonNull session: CameraCaptureSession) {
                    mPreviewSession = session
                    updatePreview()

            val realTimeOpts = FaceDetectorOptions.Builder()
            .setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
            .setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
            .enableTracking()
            .build()

        detector = FaceDetection.getClient(realTimeOpts)

 imageReader?.setOnImageAvailableListener({ reader ->
            if (isProcessing) return@setOnImageAvailableListener
         
            reader.acquireLatestImage().let { image ->
                try {
                    isProcessing = true

                    synchronized(processorLock) {
                        // Real-time contour detection

                        detector?.process(
                            InputImage.fromMediaImage(
                                image,
                                getRotationCompensation()
                            )
                        )?.addOnSuccessListener { faces ->
                            // Task completed successfully
                            // ...
                            for (face in faces) {
                                val faceGraphic = FaceGraphic(graphicOverlay, face)
                                if (faces.isNotEmpty()) {
                                Log.d(TAG, "Face detected!")
                                }
                            }
                     
                            isProcessing = false
                            image?.close()
                        }
                            ?.addOnFailureListener { e ->
                                // Task failed with an exception
                                // ...
                                image?.close()
                                detector?.close()
                                isProcessing = false                          
                                e.printStackTrace()
                            }
                }

                override fun onConfigureFailed(@NonNull session: CameraCaptureSession) {
               
                }
            }, mBackgroundHandler
        )

Anyone having Asus device can please confirm if they are facing similar issue.

  • Phone make and model: Asus Zenfone Max M2
  • OS version: Android 9
Vismay Patil
  • 64
  • 4
  • 14
  • Could you provide the full logcat? Also have you tried CameraX? CameraX is a compatibility library that takes care of device quirks. – Xi 张熹 Dec 15 '20 at 17:12
  • CameraX work fine but, I have a use-case where I am recording user's video for which I am using camera2 API. – Vismay Patil Dec 15 '20 at 17:18
  • @Xi张熹 I don't have that device with me currently please allow me some time. I shall provide you the entire logcat by the weekend. – Vismay Patil Dec 15 '20 at 17:22
  • Vismay, did you get a chance to get a logcat? We would love to solve this mystery. – Chrisito Jan 08 '21 at 19:24
  • @Chrisito the device is still not with me please allow me a few more days time shall provide you with the logcat. – Vismay Patil Jan 09 '21 at 05:33
  • @VismayPatil friendly ping :) Thanks! – Chrisito Jan 20 '21 at 18:24
  • Hi @Chrisito so sorry for delayed response. I am attaching a link of Google doc which has log please check and let me know. https://docs.google.com/document/d/1gdL4iMNeCc6twjcggIF7DNUV_4Gm_6dnAaeMsansan0/edit?usp=sharing – Vismay Patil Feb 13 '21 at 11:15
  • Hi @Chrisito please let me know if you have gone through the log file or if you need any other detail. – Vismay Patil Feb 19 '21 at 07:39
  • Could you try if CameraX sample app works on that device? So we can rule out if the camera stack is problematic. https://github.com/android/camera-samples/tree/master/CameraXBasic – Xi 张熹 Feb 19 '21 at 19:06
  • Hi, @Xi张熹 have checked the cameraX sample app and it works fine on the device. Just for your information, I am using Camera 2 API in my actual application. – Vismay Patil Feb 20 '21 at 09:17
  • Have you considered using CameraX? camera2 API sometimes needs workaround code especially on LEGACY devices such as this one. CameraX is a compatibility library that covers these corner cases. – Xi 张熹 Feb 22 '21 at 23:06
  • I have checked MLKit sample app with CameraX which works fine on that device but I also have a case where I need to record video of the user who's face is being detected the recording should not consists the cricle and contour though. – Vismay Patil Feb 23 '21 at 05:14
  • Hi @Xi张熹 I used CameraX with Preview and ImageAnalysis use cases but for the VideoCapture use case it isn't working on all the devices, the app crashes with the following error: No supported surface combination is found for camera device - Id : 1. May be attempting to bind too many use cases. Existing surfaces: [] New configs: [androidx.camera.core.impl.VideoCaptureConfig@a3a81ba, androidx.camera.core.impl.PreviewConfig@6dc6814, androidx.camera.core.impl.ImageAnalysisConfig@216cd6a] – Vismay Patil Jun 16 '21 at 10:16
  • Hi @Xi张熹 I have also added the log in the MLKit issue tracker which I raised earlier please check and let me know if anything can be done. – Vismay Patil Jun 17 '21 at 07:42
  • @VismayPatil that's correct. the VideoCapture feature is experimental and not supported on all devices. Please stay tuned for updates. To make sure VideoCapture works, you will have to disable ImageAnalysis. So to achieve what you described on older devices, you will have to use preview stream for more than one purposes. e.g. using the preview stream for preview & analysis, or using the preview stream for preview & video recording. This is not natively supported by CameraX and it will require some work. One possibility is getting the preview stream as a GL texture and "fan out". – Xi 张熹 Jun 18 '21 at 23:51
  • @Xi张熹 can you help me out here? I didn't exactly understand what "fan out" means. Also for preview stream as a GL texture are you asking to use CameraX for this? Can you provide some references? It will be helpful. – Vismay Patil Jun 21 '21 at 12:29
  • By "fan out" I mean multi purposing the output of preview, and use it for both preview and video recording. In this case, you can no longer use – Xi 张熹 Jun 21 '21 at 17:42
  • By "fan out" I mean multi purposing the output of preview, and use it for both preview and video recording. In this case, you can no longer use CameraX's VideoCapture use case. Unfortunately the task is non-trivial to do it efficiently. If you care less about the performance, you can simply use only the ImageAnalysis stream, and use it for everything. For example, use the bytes for analysis, then display the bytes as preview as well as save the bytes to a video. Still you have to figure out how to save the bytes to video without the VideoCapture use case. – Xi 张熹 Jun 21 '21 at 17:49

0 Answers0