UIAutomation missing to catch some elements

Question

I'm using the code above to walk through the descendants of a window, however, it's not catching everything like I can see using inspect.exe.

Currently testing in the chrome browser, in this current webpage I'm writing this question, it's catching just current opened pages, bookmarks, and browser buttons, it's missing to catch the page content.

While testing in other windows like a folder, for example, it does catch everything correctly, but it's taking too much time, in a folder with 120 elements it takes up to 1.4 seconds, I wonder if the code could be improved in any manner?

 #include <UIAutomation.h>
 #include <UIAutomationCore.h>
 #include <UIAutomationClient.h>

 #include "Oleacc.h"
 #include "atlbase.h"
 #pragma comment(lib,"Oleacc.lib")
 
 WCHAR window[250] = L"Ask a Question - Stack Overflow - Google Chrome";
 IUIAutomationElement *element = GetTopLevelWindowByName(window);
 ListDescendants(element, 2);
    
 IUIAutomationElement* GetTopLevelWindowByName(LPWSTR windowName)
 {
     if (windowName == NULL)
         return NULL;
    
     CComPtr<IUIAutomation> g_pAutomation;
     if (SUCCEEDED(CoInitialize(NULL)))
     {
         if (!SUCCEEDED(g_pAutomation.CoCreateInstance(CLSID_CUIAutomation8))) // or CLSID_CUIAutomation
             return NULL;
     }
    
     VARIANT varProp;
     varProp.vt = VT_BSTR;
     varProp.bstrVal = SysAllocString(windowName);
     if (varProp.bstrVal == NULL)
         return NULL;
    
     IUIAutomationElement* pRoot        = NULL;
     IUIAutomationElement* pFound       = NULL;
     IUIAutomationCondition* pCondition = NULL;
    
     // Get the desktop element. 
     HRESULT hr = g_pAutomation->GetRootElement(&pRoot);
     if (FAILED(hr) || pRoot == NULL)
         goto cleanup;
    
     // Get a top-level element by name, such as "Program Manager"
     hr = g_pAutomation->CreatePropertyCondition(UIA_NamePropertyId, varProp, &pCondition);
     if (FAILED(hr))
         goto cleanup;
    
     pRoot->FindFirst(TreeScope_Children, pCondition, &pFound);
    
 cleanup:
     if (pRoot != NULL)
         pRoot->Release();
    
     if (pCondition != NULL)
         pCondition->Release();
    
     VariantClear(&varProp);
     return pFound;
 }
    
    
    
 void ListDescendants(IUIAutomationElement* pParent, int indent)
 {
     static CComPtr<IUIAutomation> g_pAutomation;
    
     if (!g_pAutomation) {
         if (SUCCEEDED(CoInitialize(NULL)))
         {
             if (!SUCCEEDED(g_pAutomation.CoCreateInstance(CLSID_CUIAutomation8))) // or CLSID_CUIAutomation
                 return;
         }
     }
    
    
     if (pParent == NULL)
         return;
    
     IUIAutomationTreeWalker* pControlWalker = NULL;
     IUIAutomationElement* pNode = NULL;
    
     g_pAutomation->get_ControlViewWalker(&pControlWalker);
     if (pControlWalker == NULL)
         goto cleanup;
    
     pControlWalker->GetFirstChildElement(pParent, &pNode);
     if (pNode == NULL)
         goto cleanup;
    
     while (pNode)
     {
         BSTR sName;
         pNode->get_CurrentName(&sName);
    
         UIA_HWND uia_hwnd;
         pNode->get_CurrentNativeWindowHandle(&uia_hwnd);
    
         RECT rect;
         pNode->get_CurrentBoundingRectangle(&rect);
    
         ListDescendants(pNode, indent + 1);
         IUIAutomationElement* pNext;
         pControlWalker->GetNextSiblingElement(pNode, &pNext);
         pNode->Release();
         pNode = pNext;
     }
    
 cleanup:
     if (pControlWalker != NULL)
         pControlWalker->Release();
    
     if (pNode != NULL)
         pNode->Release();
    
     return;
 }

Yes, that's the way chrome works. If chrome window is not visible, then the document is not available. I suspect this way chrome is trying to save memory or something. If you maximize the Inspect tool, and click refresh, you will see the chrome window, but not the chrome document. — Barmak Shemirani, Sep 09 '21 at 17:34
Have you seen this https://stackoverflow.com/a/47227260/403671 — Simon Mourier, Sep 09 '21 at 17:37

score 0 · Accepted Answer · answered Sep 09 '21 at 19:58

Testing with Chrome version 93 and Windows 10:

If chrome window is at least partially visible, the first UIA child document in chrome is html page of the active tab, and you can walk the UIA elements in this page.

If chrome window is not visible, you cannot access the page content with UIA. You can only use UIA to access the main controls for Chrome, as explained here

You probably run the Inspect tool as small window in front of a partially visible Chrome window, so Inspect can see the page content. Otherwise there is nothing special about the Inspect tool.

Another quirk with Chrome is that when the Chrome window is in full-screen mode, the main controls are not seen by UIA. Instead only the page content is seen by UIA (as long as Chrome is not hidden behind another window)

This seems to be specific to Chrome. Firefox behaves as expected.

You're right, I tested with the window active, and it does catch the page content. Regarding the speed it takes, there's something I could improve in these functions? Even when scanning a window with few elements it's taking too much time as I wrote above. — , Sep 10 '21 at 04:09
you have to find a particular window, get its `IAutomationElement`, use `FindFirst`, `FindAll` etc to find a particular UIA element. — Barmak Shemirani, Sep 10 '21 at 16:17

UIAutomation missing to catch some elements

1 Answers1