Detecting Liveness - Eye Blinking and Smiling Probability with ML KIT Android

Question

Using Google ml-kit for face detection in my android application.

I have to also detect liveness for eye blink and smile.

Below is the function:

fun detectLiveness(image: Bitmap, face: Face): Float? {
    val aligned = this.faceSquaredAligner.computeImagePatch(face, image, 224)
    val token = this.faceSquaredAligner.computeImagePatch(face, image, 448)

    val squareData = prepareImage(aligned, 1)
    val tokenData = prepareImage(token, 1)

    var score: Float? = null
    if (face.rightEyeOpenProbability != null) {
        Toast.makeText(context,"got it", Toast.LENGTH_SHORT).show()
    }else{
        Toast.makeText(context,"Right Eye null", Toast.LENGTH_SHORT).show()
    }
    this.env.use {
        var squareTensor = OnnxTensor.createTensor(
            this.env, squareData,
            this.getTensorShape(aligned)
        )
        val tokenTensor = OnnxTensor.createTensor(
            this.env, tokenData,
            this.getTensorShape(token)
        )
        squareTensor.use {
            tokenTensor.use {
                var inputs = mutableMapOf("input.1" to tokenTensor, "input.207" to squareTensor)
                var outputs = mutableSetOf("718")
                val output = this.session.run(inputs, outputs, OrtSession.RunOptions())
                output.use {
                    @Suppress("UNCHECKED_CAST")
                    val rawOutput = ((output?.get(0)?.value) as Array<FloatArray>)[0]
                    score = softMax(rawOutput)[0]
                }
            }
        }
    }
    return score
}

In which I have checked for rightEyeOpenProbability, but getting null.

We have 'OnnxruntimeLivenessDetector' class which has method named 'detectLiveness()' which returns score (float value) This 'detectLiveness()' method is called from the method named 'processDetection()' inside 'FaceDetector' class.

At both the places I tried to check rightEye, leftEye and smiling probability but am not getting any value for it.

What am I doing wrong here? Or Is there any different way to detect liveness i.e. for user's eye blinking and smiling values? Thanks in Advance.

score 0 · Accepted Answer · answered Apr 19 '23 at 05:59

Its resolved after couple of days and it was just silly mistake. Below is the method to set face detection options you can check:

 fun getFaceDetectorOptions(context : Context?) : FaceDetectorOptions {
    val landmarkMode = FaceDetectorOptions.LANDMARK_MODE_ALL
    val contourMode = FaceDetectorOptions.CONTOUR_MODE_NONE
    //val classificationMode = FaceDetectorOptions.CLASSIFICATION_MODE_NONE
    val classificationMode = FaceDetectorOptions.CLASSIFICATION_MODE_ALL
    val performanceMode = FaceDetectorOptions.PERFORMANCE_MODE_FAST

    val enableFaceTracking = false
    val minFaceSize = "0.1".toFloat()
    val optionsBuilder = FaceDetectorOptions.Builder()
        .setLandmarkMode(landmarkMode)
        .setContourMode(contourMode)
        .setClassificationMode(classificationMode)
        .setPerformanceMode(performanceMode)
        .setMinFaceSize(minFaceSize)
    if (enableFaceTracking) {
        optionsBuilder.enableTracking()
    }
    return optionsBuilder.build()
}

You can see their I have commented val classificationMode = FaceDetectorOptions.CLASSIFICATION_MODE_NONE and instead used val classificationMode = FaceDetectorOptions.CLASSIFICATION_MODE_ALL and it simply resolved this issue.

Detecting Liveness - Eye Blinking and Smiling Probability with ML KIT Android

1 Answers1