2

Goal: I want to retrieve the following attributes of a UI element: Name, Class, Control type and parent name. The element could belong to Outlook or web application which is opened on Internet explorer.

Python interpreter: 3.7

Python library: Comtypes

Research made:

OUTLOOK: In order to retrieve the attributes of a particular element in Microsoft Outlook, I am using comtypes to create an object of UIautomationCore.dll. I could retrieve the name and class based on the point of a particular element, but I cannot find a way to retrieve the control type and on the other hand the parent element name having this point. I found in the Microsoft UI Automation that I must use the TreeWalker in order to get the information about the parent element. However, I do not know how to implement in Python.

WEB APPLICATION (Internet explorer): I am using the same library and I can retrieve the elements which are located in the Internet Explorer browser. However, when I cannot get these attributes of one button which are located in the content of the web application (body).

Question: How to retrieve the control type and parent element name in Outlook with Microsoft UI automation? and How to retrieve those attributes when I am working with Internet Explorer?.

My current code:

import comtypes
from comtypes.client import *

comtypes.client.GetModule('UIAutomationCore.dll')
from comtypes.gen.UIAutomationClient import *

# get IUIAutomation interface
uia = CreateObject(CUIAutomation._reg_clsid_, interface=IUIAutomation)


# import tagPOINT from wintypes
from ctypes.wintypes import tagPOINT
point = tagPOINT(1833, 95)
element = uia.ElementFromPoint(point)

walker = ViewWalker
parentElement = walker.GetParent(element)
name=element.currentName
parentName = parentElement.currentName

print("elementName:", name)
print("parentElementName: ", parentName)

Favor: Could someone give some guidelines on how to implement IUIAutomationElement COM interface with Python?. Any link or book could be very useful to understand it.

sergioMoreno
  • 189
  • 1
  • 15

2 Answers2

1

The module pywinauto has this kind of functionality (basically wrapping up/ abstracting all this COM stuff you want to do to give you access to the UIAutomationElement information). For the basics: https://pywinauto.readthedocs.io/

However, ff you're interested in the finicky details of how it's done using the COM interface with python, their code and docs would be a good starting point!

Julia Rozanova
  • 11
  • 1
  • Hi @Julia Rozanova. Tks for your help and time. As far as I understood with pywinauto is not possible to get the body of a website. If so, how could I achieve it? – sergioMoreno Jul 14 '20 at 08:15
0

There are libraries for this:

my favorite, uiautomation. There is an inspector and the library is very complete. Although it doesn't work 100% well with IE and Java applications.

https://pypi.org/project/uiautomation/

With this you can get the selector of each windows element, by placing the mouse over it

from uiautomation import uiautomation as auto

while True:
    control = auto.ControlFromCursor()
    print("Name", control.Name)
    print("AutomationID", control.AutomationId)

Another library is pywinauto, but I have not used it much, therefore, I can not tell you more

Danilo Toro
  • 569
  • 2
  • 15