I am working on an automation project, in which I need to capture the activities [ application launched, data entered, input type etc.] user performs on a desktop. I came across Microsoft UI Automation framework which so far works well for native windows based applications like MS Office, .NET apps etc. However I did not find any useful information / samples of capturing the information from different web browsers [Chrome is a must], Python apps, Java Apps etc. Can someone please confirm whether MS UI Automation Framework supports such apps. Any working example to extract user activities from these apps would be highly appreciated. Thanks.
Asked
Active
Viewed 5,038 times
3 Answers
9
Chrome only supports UI Automation for toolbars, tabs, menu, buttons around the web page. Everything that's rendered as a web page is not seen by UIA.
For the web page content, the easiest way is to use Selenium (driven by the ChromeDriver), which is kind of a de facto standard for browsers, and has nothing to do with UIA.
To test if an app supports UIA, and how far it does, it's very easy, just run UIA's Inspect tool and check the UI tree over that application.
Simon Mourier
- 132,049
- 21
- 248
- 298
-
UIA should see the chrome document, as long as the document is at least partially visible. You can test this with Inspect tool on a partially visible chrome window, vs minimized chrome window. This is in the year 2021. – Barmak Shemirani Sep 09 '21 at 17:48
-
@BarmakShemirani - some (!) answers are dated on SO yes, but not everybody uses 2021 technologies, why don't you make your own answer? – Simon Mourier Sep 09 '21 at 18:06
-
This answer is correct except for odd behavior of Chrome. Firefox for example works as expected. – Barmak Shemirani Sep 09 '21 at 20:01
9
Some additions to Simon's answer...
Chrome page content can be seen by UIA if you run chrome --force-renderer-accessibility. Only for existing Chrome process it won't work. Though user can create a new tab chrome://accessibility manually and enable UIA for all or some chosen pages. This method also works for AT-SPI accessibility technology on Linux. Of course, Selenium WebDriver is an industry standard here. But another way exists. Both Mozilla and IE support UIA by default.
Inspect.exe can be simply downloaded from this GitHub repo.
Regarding Java apps it depends on the app type. Your chances is about 50/50.
WxPython or PyQt5 are good for UIA. TkInter or Kivy apps are not.
P.S. There is an example how to drag a file from explorer.exe and drop to Google Drive in Chrome using Python library pywinauto.
Vasily Ryabov
- 9,386
- 6
- 25
- 78
-
Need some help. You say: "Chrome page content can be seen by UIA". I want to get events performed on a webapp on chrome (like keep track of keys pressed and button clicked and on what element). Can I do this by using UIA on chrome?. – grit639 Aug 08 '21 at 04:41
-
Could you please help me with: https://stackoverflow.com/questions/68703414/setwineventhook-has-event-object-invoked-but-event-is-not-triggered-on-clicking – grit639 Aug 08 '21 at 18:16
-
@grit639 THIS. THIS HELPED ME SO, SO MUCH!!! In order to `Force` Chrome to allow your application to see it, you have to call Chrome from a Command Line such as follows: ``Start "Chrome" Chrome.Exe --force-renderer-accessibility "https://yourURLhere.com"`` and it will allow your application to view the objects using UIA. An important note though! It must open a new window with this flag, and using ``--new-window`` does _NOT_ work - I just confirmed. You must open a completely new instance of Chrome for it to manifest. – k1dfr0std Feb 21 '22 at 03:40
4
I'm a bit late to the party.. But Chromes accessibility features are only activated once something tries to access it's accessibility.
If you call AccessibleObjectFromWindow ([DllImport("oleacc.dll")]) with the window handle an existing chrome window will have its accessibility activated (and you'll see the actual web page content in UIA!).
If the chrome window is opened after your app is running - Chrome pings open processes for any open accessibility apps... for that you use AccessibleObjectFromEvent and the event you're responding to comes from the windows pipeline: EVENT_SYSTEM_ALERT = 0x0002 .
The bottom line is - you have to tell chrome that there's something installed that wants to access it's web page content.
Oh! and your application has to be signed!! Unsigned apps won't be able to access web content - I think that's the same in firefox too.
I hope this helps someone in the future.
See: https://www.chromium.org/developers/design-documents/accessibility
-
Could you please help me with : https://stackoverflow.com/questions/68703414/setwineventhook-has-event-object-invoked-but-event-is-not-triggered-on-clicking – grit639 Aug 08 '21 at 18:17
-
This is good to know as well. This would be the `legit` way to build an application I suppose, vs the answer I'm currently using where I force Chrome open w/ the accessibility flag turned on – k1dfr0std Feb 21 '22 at 03:47