Detect x y coordinates of specific text

Question

I tried write an automation for my games in Android with Tasker and AutoTools plugins. Somethings ok at this point but i need capture the screenshot and need interpret it for my needs.

That's exactly what I need;

Some texts are important in games and i want to click on it wherever they are in the screen. So i need OCR for this task i think. I follow some solutions but fail or stuck everytime. Let me explain which solutions i tried.

Following Solution 1:

I tried AutoInput (Tasker plugin) UIQuery method but fail. Because UIQuery of AutoInput just works on android UI i think. Cant get any information from 3D App like games.

Following Solution 2:

I search OCR solution and find AutoTools (Tasker plugin)
Create a task and take screenshot and interpret it with AutoTools OCR method. Thats ok. AutoTools OCR succesfully read a text from Image file.
But i stuck again. Because i succesfully read a text from image file but i dont know x y coordinate of important text.

What suggest at this point?

Should i learn android and write own app?

These are not the Autotools you're looking for. (The [autotools] tag refers to some of the key components of the GNU build system. As far as I know, there is no tag for any OCR package going by that name. Tags edited.) — John Bollinger, Dec 21 '18 at 14:10

score 2 · Answer 1 · answered Dec 20 '18 at 19:21

You should checkout out the ocr-reader Google sample. It's quick to run and not too difficult to get what you're looking for. What you would need to do is modify the OcrDetectorProcess that comes with the sample to break down the text into individual words, then you can easily calculate the boundaries and center points of each word. Here's some code to get you started:

@Override
public void receiveDetections(Detector.Detections<TextBlock> detections) {
    mGraphicOverlay.clear();

    // Get all detected items.
    SparseArray<TextBlock> items = detections.getDetectedItems();
    for (int i = 0; i < items.size(); ++i) {
        TextBlock item = items.valueAt(i);

        // Get individual lines in each item.
        List<Line> lines = (List<Line>) item.getComponents();
        for (Line line : lines) {

            // Get individual "words" in each line.
            List<Element> elements = (List<Element>) line.getComponents();
            for (Element e : elements) {

                // Now get the position of each element.
                Rect rect = e.getBoundingBox();
                Point[] points = e.getCornerPoints();
                int centerX = (points[0].x + points[2].x) / 2;
                int centerY = (points[0].y + points[2].y) / 2;

                // DO STUFF

            }
        }
    }
}

Thanks for your interest first Danny. Looks like a Java code. Did you suggest learn Android basics and write own app right? — Lacrymae, Dec 20 '18 at 19:37

score 1 · Accepted Answer · answered Dec 21 '18 at 20:30

I contact with the developer who write the "AutoTools" Tasker plugin.

He/She add some function to plugin and solve it.

Plugin, interpret with OCR granted image and return words and center of xy positions of each words now.

If anyone search like this function for Android and Tasker App please visit this forum topic link. Its very useful.

Detect x y coordinates of specific text

2 Answers2