ML Kit Barcode Scanning doesn't detect QR codes in the photo of the monitor screen

Question

I am using com.google.mlkit:barcode-scanning:17.0.2 to detect QR codes in the pictures.

After getting URI from the gallery I create InputImage and then process this image with BarcodeScanner to find QR codes. When I select a photo of QR codes on paper code is found. But when I take a photo of the QR code on the monitor screen code is never found. What I should do to be able to detect a QR code in a photo of a monitor screen?

(When I use the same scanner with CameraX to do live QR code detection it finds code on the monitor screen)

val image = InputImage.fromFilePath(context, uri)

val scanOptions =
    BarcodeScannerOptions.Builder()
        .setBarcodeFormats(
            Barcode.FORMAT_QR_CODE,
        )
        .build()

val scanner = BarcodeScanning.getClient(scanOptions)

scanner.process(image)
    .addOnSuccessListener {
        val code = it.getOrNull(0)?.rawValue
        if (code == null) {
            // code NOT found
        } else {
            // code was found
        }
    }

Example of QR code on paper which is found

Example of QR code on the monitor screen which is NOT found

When you took a photo of the monitor screen, what is the resolution used? You can increase the resolution to have a better detection results. — Shiyu, Apr 01 '22 at 17:44
Native resolution of the camera so very high. I was testing this picture and I noticed that when I decrease photo resolution to e.g. 216x384 code will be found. But when picture is in 1080x1920 code won't be found. So it looks like big resolution is a problem becouse in the monitor photo You can notice something like "waves of pixels", maybe this is problem — iknow, Apr 01 '22 at 17:49
I´ve had this issue too. It can be related with low light. Try increasing the light around the monitor - in my experience it helps a lot. — Curious Mind, Apr 09 '22 at 15:23

LSerni · Answer 1 · 2023-01-27T09:27:52.337

Chances are that you're fighting against Moiré effect. Depending on the QR detection algorithm, the high frequencies introduced by the Moiré effect can throw the detector off its track. Frustratingly, it is often the better QRcode detectors that are defeated by Moiré patterns.

A good workaround is:

take the picture at the highest resolution you can
perform a blurring of the picture
increase contrast to the max, if possible
(optionally) run a sigma thresholding, or just rewrite all pixels with a luma component below 32 to 0, all those above 224 to 255.

Another way of doing approximately the same operation is

take the picture at the highest resolution you can
increase contrast to the max, if possible
downsample the picture to a resolution which is way lower

The second method gives worse results, but usually can be implemented with device primitives.

Another source of problems with monitors (not in your picture as far as I can see) is the refresh rate. Sometimes, you'll find that the QR code is actually an overexposed QRcode in the upper half of the picture and an underexposed QRcode in the bottom half of the picture. Neither are recognized. This effect is due to the monitor's refresh rate and strategy and is not easy to solve - you can try lowering the monitor's luminosity to increase exposure time, until it exceeds 1/50th or 1/25th of a second, or take the picture from farther away and use digital zooming. Modern monitors have higher refresh rates and actually refresh at more than their own dwell time, so this should not happen; with old analog monitors however it will happen every time.

A third, crazy way

This was discovered half by chance, but it works really well even on cheap hardware provided the QR SDK or library supplies some small extra frills.

Take a video of about 1 second length at the highest frame rate you can get (25 fps?).
From the middle (e.g. 13th) frame, extract the three QR "waypoints" - there might be a low-level function in your SDK called "containsQRCode()" that does this. If it returns true, the waypoints were found and their coordinates are returned to allow performing scaling/estimates. It might return a confidence figure ("this picture seems to contain a QR code with probability X%"). These are the APIs used by apps to show a frame or red dots around candidate QR codes. If your SDK doesn't have these APIs, sorry... you're out of luck.
Get the frames immediately before and after (12th and 14th), then the 11th and 15th, and so on. If any of these returns a valid QR code, you're home free.
If the QR code is found (even if not correctly decoded) in enough frames, but the waypoint coordinates vary much, the hand is not steady - say so to the user.
If you have enough frames with coordinates that vary little, you can center and align on those, and average the frames. Then run the real QRCode recognition on the resulting image. This gets rid of 100% of the Moiré effect, and also drastically reduces monitor dwell noise with next to no information loss. The results are way better than the resolution change, which isn't easy to perform on (some) devices that reset the camera upon resolution change.

This worked on a $19 ESP32 IoT device operating in a noisy, vibration-rich environment (it acquires QR codes from a camera image of carton boxes on a moving transport ribbon).

Not the answer with a full implementation that I would expect but well explained. Thanks! I was doing it in a second way, it works well with QR codes but not so good with really long barcodes which have to be of good quality. — iknow, Apr 10 '22 at 07:36

ML Kit Barcode Scanning doesn't detect QR codes in the photo of the monitor screen

Example of QR code on paper which is found

Example of QR code on the monitor screen which is NOT found

1 Answers1

A third, crazy way