Android MLKit face detection not detecting faces when using Bitmap

Question

I have an XR app, where display shows the camera (rear) feed. As such, capturing the screen is pretty much the same as capturing the camera feed... As such, I take screenshots (Bitmaps) and then try to detect faces within them using Googles MLKit.

I'm following the official guide to detect faces.

To do this, I first init my face detector:

FaceDetector detector;

public MyFaceDetector(){
    FaceDetectorOptions realTimeOpts =
            new FaceDetectorOptions.Builder()
                    .setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
                    .build();
    detector = FaceDetection.getClient(realTimeOpts);
}

I then have a function which passes in a bitmap. I first convert the bitmap to a byte array. I do this because InputImage.fromBitmap is very slow, and MLKit actually tells me that I should use a byte array:

ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.JPEG, 85, byteArrayOutputStream);
byte[] byteArray = byteArrayOutputStream .toByteArray();

Next I make a mutable copy of the Bitmap (so that I can draw onto it), and set up a Canvas object, along with a color that will be used when drawing on to the Bitmap:

BitmapFactory.Options options = new BitmapFactory.Options();
options.inMutable = true;
Bitmap bmp = BitmapFactory.decodeByteArray(byteArray, 0, byteArray.length, options);
Canvas canvas = new Canvas(bmp);
Paint p = new Paint();
p.setColor(Color.RED);

After all is set up, I create an InputImage (used by the FaceDetector), using the byte array:

InputImage image = InputImage.fromByteArray(byteArray, bmp.getWidth(), bmp.getHeight(),0, InputImage.IMAGE_FORMAT_NV21);

Note the image format... There is a InputImage.IMAGE_FORMAT_BITMAP, but using this throws an IllegalArgumentException. Anyway, I next try to process the Bitmap, detect faces, fill each detected face with the color defined earlier, and then save the Bitmap to disk:

Task<List<Face>> result = detector.process(image).addOnSuccessListener(
            new OnSuccessListener<List<Face>>() {
                @Override
                public void onSuccess(List<Face> faces) {
                    Log.e("FACE DETECTION APP", "NUMBER OF FACES: " + faces.size());

                    Thread processor = new Thread(new Runnable() {
                        @Override
                        public void run() {
                            for (Face face : faces) {
                                Rect destinationRect = face.getBoundingBox();
                                canvas.drawRect(destinationRect, p);
                                canvas.save();
                                Log.e("FACE DETECTION APP", "WE GOT SOME FACCES!!!");

                            }
                            File file = new File(someFilePath);
                            try {
                                FileOutputStream fOut = new FileOutputStream(file);
                                bmp.compress(Bitmap.CompressFormat.JPEG, 85, fOut);
                                fOut.flush();
                                fOut.close();
                            } catch (Exception e) {
                                e.printStackTrace();
                            }
                        }
                    });

                    processor.start();
                }
            })
            .addOnFailureListener(
                    new OnFailureListener() {
                        @Override
                        public void onFailure(@NonNull Exception e) {
                            // Task failed with an exception
                            // ...
                        }
                    });
}

While this code runs (i.e. no exceptions) and the bitmap is correctly written to disk, no faces are ever detected (faces.size() is always 0). I've tried rotating the image. I've tried changing the quality of the Bitmap. I've tried with and without the thread to process any detected faces. I've tried everything I can think of.

Anyone have any ideas?

ML Kit InputImage. fromByteArray only support yv12 and nv21 formats. You will need to convert the bitmap to one of these formats in order for ML kit pipeline to process. Also, if the original image you have is a bitmap, you can probably just use InputImage.fromBitmap to construct an InputImage. It shouldn't be slower than your current approach. — Chenxi Song, Jun 14 '21 at 17:56

score 1 · Answer 1 · answered Jun 18 '21 at 07:12

ML Kit InputImage. fromByteArray only support yv12 and nv21 formats. You will need to convert the bitmap to one of these formats in order for ML kit pipeline to process. Also, if the original image you have is a bitmap, you can probably just use InputImage.fromBitmap to construct an InputImage. It shouldn't be slower than your current approach.

score 0 · Answer 2 · answered Jan 05 '22 at 15:42

I was having the same issue use ImageInput.fromMediaImage(..., ...)

override fun analyze(image: ImageProxy) {

     val mediaImage: Image = image.image.takeIf { it != null } ?: run {
                image.close()
                return
            }
     val inputImage = InputImage.fromMediaImage(mediaImage, image.imageInfo.rotationDegrees)
     // TODO: Your ML Code
}

Check here for more details https://developers.google.com/ml-kit/vision/image-labeling/android

Android MLKit face detection not detecting faces when using Bitmap

2 Answers2