Text Recognition not accurate on Android device using camera and Firebase ML Kit

Question

I am using Firebase ML Kit on Android device for text recognition using a camera without clicking image. I am using it by receiving frames and getting bitmaps from the frames. Then passing the Bitmaps into the Text Recognition method. But the text recognized is not accurate. Also, it is constantly changing but never giving accurate results. Please let me know what I am doing wrong.

getting frames and Bitmaps:

  public void onSurfaceTextureUpdated(SurfaceTexture surface) {
            frame = Bitmap.createBitmap(textureView.getWidth(), textureView.getHeight(), Bitmap.Config.ARGB_8888);
            textureView.getBitmap(frame);

            Bitmap emptyBitmap = Bitmap.createBitmap(textureView.getBitmap(frame).getWidth(), textureView.getBitmap(frame).getHeight(), textureView.getBitmap(frame).getConfig());
            if (textureView.getBitmap(frame).sameAs(emptyBitmap)) {
                // myBitmap is empty/blank
                System.out.println(" empty !!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
            } else {
                System.out.println(" bitmap");


                        bitmap = textureView.getBitmap(frame);
                                runTextRecognition();
            }

text recognition:

private void runTextRecognition() {
    System.out.println(" text recognition!!!");
    FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);
    FirebaseVisionTextRecognizer recognizer = FirebaseVision.getInstance().getOnDeviceTextRecognizer();
    recognizer.processImage(image).addOnSuccessListener(new OnSuccessListener<FirebaseVisionText>() {
        @Override
        public void onSuccess(FirebaseVisionText texts) {
            System.out.println("Text recognized ::: " + texts);
            textRecognized = true;
            processTextRecognitionResult(texts);

        }
    }).addOnFailureListener(new OnFailureListener() {
        @Override
        public void onFailure(@NonNull Exception e) {
            e.printStackTrace();
        }
    });

}

It might be an issue with the Bitmap you are creating. Can you load a static Bitmap, with known size and format, and see if that works? — Sachin K, Jul 08 '19 at 11:56
I tried it, that also didn't work as the type of text was MRZ string. — AroshiS, Jul 30 '19 at 09:38

score 3 · Accepted Answer · answered Jul 30 '19 at 09:37

3

The text I was trying to recognize was MRZ. I contacted Firebase Support, they themselves performed tests and concluded that ML Kit API isn't capable of reading MRZ type text and that they might incorporate it in future.

answered Jul 30 '19 at 09:37

AroshiS

79
1
9

score 1 · Answer 2 · answered Jul 08 '19 at 08:58

1

You can try Mobile Text Vision API for OCR (Optical Character Recognition) in Android.

Refer to this Google code lab for implementation details https://codelabs.developers.google.com/codelabs/mobile-vision-ocr/index.html?index=..%2F..index#0

Especially creating OcrDetectorProcessor step.

answered Jul 08 '19 at 08:58

TheAnkush

885
7
10

I have tried this, but the results are not accurate using this too. – AroshiS Jul 08 '19 at 09:33
can we tweak Firebase ML kit a little bit to get 100% accuracy ? – AroshiS Jul 08 '19 at 09:34
@AroshiS I'm too lazy to check the actual provided public API, but when I was using tesseract library directly, I could have set up legal alphabet, for MRZ reading it did help to clean up the results a bit, setting up only the legal characters. Also MRZ does use special font and is monotype/space, I could pre-set the font type in Tesseract, and I did calculate character boundary boxes afterward from the result, to verify how well they fit into the grid vs their expected size (like ">" can't be 1/4 of size of "A", etc...) ... that helped me to filter noise results even a bit more. – Ped7g Jul 30 '19 at 09:54
I.e. check how much configuration you have over the local Vision API (if it is still available), it used to be based on Tesseract, but I think the implementation may have changed to NN meanwhile, and not sure how rich the API is.. – Ped7g Jul 30 '19 at 09:55
@Ped7g thank you so much for your answer. Can you please share some source code for more understanding of what you are trying to say? – AroshiS Aug 01 '19 at 05:55
@AroshiS unfortunately it's commercial work under closed source license, not allowing me to post larger parts of it and I wrote it a bit as "proof of concept", never managed to truly clean it up, so I would have difficult times to post only some small part, which would be meaningful. I honestly believe sharing only the idea with you is actually for your own good... :D ... sometimes inspiration works miracles... – Ped7g Aug 01 '19 at 08:22

Text Recognition not accurate on Android device using camera and Firebase ML Kit

2 Answers2