Why is Android View Client returning the view above the view I'm trying to select?

Question

(Edit: the following problem only seems to occur on my Samsung S10e. I just got home and tried Android View Client on a Pixel phone and I'm not having any problem.)

I'm automating the input process for a jetlag calculator android app Entrain. Using the culebra tool, I can find each of the views on the screen. But when I use the ViewClient.touch() method on a view, the program touches the view immediately above it on the screen.

For example, culebra lists these 2 views as the last views on the main menu of the app:

edu_umich_entrain___id_mainTableItem = vc.findViewWithTextOrRaise(u'Update Settings')
edu_umich_entrain___id_mainTableItem = vc.findViewWithTextOrRaise(u'Extras')

So I wrote this bit of code to touch the 'Extras' item:

vc = ViewClient(device, serialno)
vc.findViewWithTextOrRaise(u"Extras", root = 'ROOT').touch()

But it touches the 'Update Settings' view. (The same is true for any other view I touch.)

I've tried finding views with the findViewByIdOrRaise() method but I get the same result. I've tried the culebra GUI tool but it freezes when I click anywhere on the GUI.

I've written similar scripts to automate other android apps (Lutron, IAquaLink, Ankidroid) and they all work fine.

The coordinates for the 2 views are not overlapping, as revealed by getCoords:

print(vc.findViewWithTextOrRaise(u'Update Settings').getCoords())
print(vc.findViewWithTextOrRaise(u'Extras').getCoords())

((195, 1380), (945, 1566))
((195, 1575), (945, 1761))

If I calculate and touch the center of the 'Extras' view:

coords = vc.findViewWithTextOrRaise(u'Extras').getCoords()
x = (coords[1][0] + coords[0][0]) / 2
y = (coords[1][1] + coords[0][1]) / 2
vc.touch(x, y)

it still touches the higher 'Update Settings' view.

If I add 100 to the y axis of the center point, it finally touches the correct view. For now, I'll do that as a workaround.

coords = vc.findViewWithTextOrRaise(u'Extras').getCoords()
x = (coords[1][0] + coords[0][0]) / 2
y = (coords[1][1] + coords[0][1]) / 2 + 100
vc.touch(x, y)

Try to determine if there are overlapping views. `touch()` touches the center of the view. Use `edu_umich_entrain___id_mainTableItem.getCoords()` to check the coordinates of the view and compare with others. — Diego Torres Milano, Aug 19 '19 at 04:38
That sounds promising. Can you help me create a View instance? I tried: v = View('edu_umich_entrain___id_mainTableItem' , device) but I get this error (TypeError: string indices must be integers, not str) when I try to print(v.getCoords()) . — Rivers Cuomo, Aug 19 '19 at 05:17
Thank you. I updated the question with the results of getCoords(). In short, the 2 views do not appear to be overlapping. — Rivers Cuomo, Aug 19 '19 at 11:12
Is this the app? https://play.google.com/store/apps/details?id=edu.umich.entrain&hl=en_US — Diego Torres Milano, Aug 20 '19 at 00:24
Yes. I put a link to the app in my original question. I also realized that the problem may be specific to my model of phone, the Samsung S10e, because it's not happening on a Pixel phone. — Rivers Cuomo, Aug 20 '19 at 04:33

score 1 · Accepted Answer · answered Aug 20 '19 at 06:15

Unfortunately, I don't have a Samsung S10e to test.

Nonetheless, you may find these tricks useful (I used a Pixel).

The bounds of the views can be obtained running

$ dump --bounds

for example

      android.widget.LinearLayout   ((0, 1210), (1080, 1381))
         android.widget.ImageView edu.umich.entrain:id/img  ((0, 1210), (171, 1381))
         android.widget.TextView edu.umich.entrain:id/mainTableItem Update Settings ((171, 1210), (827, 1362))
         android.widget.ImageView edu.umich.entrain:id/img2  ((827, 1210), (958, 1341))
      android.widget.LinearLayout   ((0, 1381), (1080, 1552))
         android.widget.ImageView edu.umich.entrain:id/img  ((0, 1381), (171, 1552))
         android.widget.TextView edu.umich.entrain:id/mainTableItem Extras ((171, 1381), (827, 1533))
         android.widget.ImageView edu.umich.entrain:id/img2  ((827, 1381), (958, 1512))

If you want to actually see every view individually you can use

$ mkdir imgs
$ dump --save-view-screenshots=$PWD/imgs/

and when finished, the directory contains a screenshot of each view, for example

and

this way you can determine if there's an offset or something else in the bounds obtained by AVC/culebra.

Lastly, if for some reason you have to apply a workaround that depends on your device model or vendor, you can do something like

...
device, serialno = ViewClient.connectToDeviceOrExit(**kwargs1)
...
if device.getProperty('ro.product.vendor.model') == 'Pixel':
   # do something for pixel
   ...

Hope this helps.

Yes, there must be some kind of offset on the S10e. I took screenshots of the same views on both phones with the "dump --save-view-screenshots" tool as you suggested. The Pixel images were perfectly centered on buttons, etc. But the S10e images were off the mark, often some unusable chunk of the background image. — Rivers Cuomo, Aug 20 '19 at 19:16

Why is Android View Client returning the view above the view I'm trying to select?

1 Answers1