Get text content of the android screen

Question

Is there any way to get the text content that is being displayed on the android mobile screen. It may be by any android app.

for example in the image linked above, i wish to get all the text (starting from the "Larry Page" till the end "Fred Smith" and also the position of all the words on the screen) that is in the image and its respective position on the screen.

@phuc_tran seems like he wants to scratch the text from any android app. — aga, Jun 18 '15 at 08:31
No, the text won't be from my app, i want to acquire any and every text that is being displayed by any app on the mobile screen. — zeekhuge, Jun 18 '15 at 14:38
How this app is doing, though! https://play.google.com/store/apps/details?id=com.camel.corp.universalcopy — Madhav Anadkat, Dec 24 '16 at 13:07

score 16 · Answer 1 · edited Apr 14 '19 at 23:37

This can be achieved using the Accessibility feature of Android (as the Universal copy app does).

What you need to do is implement a class called AccessabilityService which can access all the screen UI through a listener.

In order to that you need to declare it in the manifest:

 <service android:name=".MyAccessibilityService">
 <intent-filter>
     <action android:name="android.accessibilityservice.AccessibilityService" />
 </intent-filter>
 <meta-data android:name="android.accessibilityservice" android:resource="@xml/accessibilityservice" />

the file @xml/accessibilityservice is where you configure the service like to which package to listen or what kind of events you want to get:

<accessibility-service
 android:accessibilityEventTypes="typeViewClicked|typeViewFocused"
 android:packageNames="com.example.android.myFirstApp, com.example.android.mySecondApp"
 android:accessibilityFeedbackType="feedbackSpoken"
 android:notificationTimeout="100"
 android:settingsActivity="com.example.android.apis.accessibility.TestBackActivity"
 android:canRetrieveWindowContent="true"/>

In the class override the method:

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
    Log.v(TAG, String.format(
            "onAccessibilityEvent: type = [ %s ], class = [ %s ], package = [ %s ], time = [ %s ], text = [ %s ]",
            getEventType(event), event.getClassName(), event.getPackageName(),
            event.getEventTime(), getEventText(event)));
}

In that method, you should find in the UI the desired text/TextView/EditText and extract from it the text you want.

NOTE: this way required the Accessibility permission which only the user can give you and it cannot be done in runtime.

You can read more in the Android documentation about how to implement an AccessabilityService

EDIT: after the question in the comment I'm adding the code to read from all the text components in the screen:

    private void printAllViews(AccessibilityNodeInfo mNodeInfo) {
        if (mNodeInfo == null) return;
        String log ="";
        for (int i = 0; i < mDebugDepth; i++) {
            log += ".";
        }
        log+="("+mNodeInfo.getText() +" <-- "+ 
        mNodeInfo.getViewIdResourceName()+")";
        Log.d(TAG, log);
        if (mNodeInfo.getChildCount() < 1) return;
        mDebugDepth++;

        for (int i = 0; i < mNodeInfo.getChildCount(); i++) {
            printAllViews(mNodeInfo.getChild(i));
        }
        mDebugDepth--;
    }

Just don't forget to initialize the AccessibilityNodeInfo and the depth counter like so:

public void onAccessibilityEvent(AccessibilityEvent event) {
    mDebugDepth = 0; 
    mNodeInfo = event.getSource();
    printAllViews(mNodeInfo);
 ...
}

Hello. I tried to log like you did but cannot get any text on screen. Can you share more about your work? Thanks. — Harry T., Jul 09 '18 at 10:32
Do i need to manually create the accessibility services xml? In which folder? — Daniel Vilas-Boas, Feb 20 '19 at 18:26
yes, you need to create that file manually. the path is: /app/src/main/res/xml/accessibilityservice.xml — Yosef, Feb 24 '19 at 06:49
Why do you use `mDebugDepth`? Anything more to take care instead of only for logging? Because sometimes i get `java.lang.StackOverflowError: stack size 8MB` after 10 calls `printAllViews()` ~ `mDebugDepth` == 10 ?! — Harry T., Jun 05 '19 at 19:02

score 2 · Answer 2 · answered Feb 23 '18 at 01:46

This can be done by using "uiautomatorviewer". This bin file can be found inside Android SDK/tools/bin/ folder.

Run the uiautomatorviewer in the terminal.

$./uiautomatorviewer

Make sure USB debugging is enabled and connect the device to PC.
Click on uiautomator dump button
Click on the text and you can copy the text from "Node details" on the side.

Incase of webview, the above approach might not work.

If you want to do it programmatically, do check at android.getText() method

Get text content of the android screen

2 Answers2

Linked