Detect Eye Blink Using Google Vision API

Question

i'm using the vision API for face detection, now i want to implement eye blink but still vision api detect eye while one eye is off.

please help me how to implement eye blink feature.

Answer 1

The "eye open probability" values from the face are the key to detecting the blink. In addition, you can use a Tracker to keep track of the eye state over time, to detect the sequence of events that indicate a blink:

both eyes open -> both eyes closed -> both eyes open

Here's an example tracker:

public class BlinkTracker extends Tracker<Face> {
  private final float OPEN_THRESHOLD = 0.85;
  private final float CLOSE_THRESHOLD = 0.15;

  private int state = 0;

  public void onUpdate(Detector.Detections<Face> detections, Face face) {
    float left = face.getIsLeftEyeOpenProbability();
    float right = face.getIsRightEyeOpenProbability();
    if ((left == Face.UNCOMPUTED_PROBABILITY) ||
        (right == Face.UNCOMPUTED_PROBABILITY)) {
      // At least one of the eyes was not detected.
      return;
    }

    switch (state) {
      case 0:
        if ((left > OPEN_THRESHOLD) && (right > OPEN_THRESHOLD)) {
          // Both eyes are initially open
          state = 1;
        }
        break;

        case 1:
          if ((left < CLOSE_THRESHOLD) && (right < CLOSE_THRESHOLD)) {
            // Both eyes become closed
            state = 2;
          }
          break;

        case 2:
          if ((left > OPEN_THRESHOLD) && (right > OPEN_THRESHOLD)) {
            // Both eyes are open again
            Log.i("BlinkTracker", "blink occurred!");
            state = 0;
          }
        break;
    }
  }

}

Note that you need to also enable "classifications" in order to have the detector indicate if eyes are open/closed:

FaceDetector detector = new FaceDetector.Builder(context)
    .setClassificationType(FaceDetector.ALL_CLASSIFICATIONS)
    .build();

The tracker is then added as a processor for receiving face updates over time from the detector. For example, this configuration would be used to track whether the largest face in view has blinked:

detector.setProcessor(
    new LargestFaceFocusingProcessor(detector, new BlinkTracker()));

Alternatively, you could use a MultiProcessor instead of LargestFaceFocusingProcessor if you are interested in detecting blinks for all faces (not just the largest face).

Answer 2

This answer assumes that you already have the code to detect the face(s) running.

The Face class has function float getIsLeftEyeOpenProbability() and float getIsRightEyeOpenProbability(), You can use them on every frame to find whether one of the eyes was blinked, if one of the function returns a large value and the other returns a smaller value.

Official documentation of the Face class can be found here

Answer 3

here is my implentation fo eye detector application using FaceDetectorAPi() whose accuracy is more than 90% when tested

private double leftEyeOpenProbability = -1.0;
private double rightEyeOpenProbability = -1.0;

private boolean isEyeBlinked(){

    if(mFaces.size()==0)
        return false;

    Face face = mFaces.valueAt(0);
    float currentLeftEyeOpenProbability = face.getIsLeftEyeOpenProbability();
    float currentRightEyeOpenProbability = face.getIsRightEyeOpenProbability();
    if(currentLeftEyeOpenProbability== -1.0 || currentRightEyeOpenProbability == -1.0){
        return false;
    }

    if(leftEyeOpenProbability>0.9 || rightEyeOpenProbability > 0.9){
        boolean blinked = false;
        if(currentLeftEyeOpenProbability<0.6 || rightEyeOpenProbability< 0.6){
            blinked = true;
        }
        leftEyeOpenProbability = currentLeftEyeOpenProbability;
        rightEyeOpenProbability = currentRightEyeOpenProbability;
        return blinked;
    }else{
        leftEyeOpenProbability = currentLeftEyeOpenProbability;
        rightEyeOpenProbability = currentRightEyeOpenProbability;
        return false;
    }
}