2

I need to retrieve the index of the caret inside a textbox in the focused window, maybe using UI Automation, or maybe a Win32 API function, if there's any function that cat do that. And I emphasize, I don't mean the x,y coordinates, but the index of the caret inside the text of the textbox. How can I do that? Also see this similar question.

Tal XD
  • 45
  • 6
  • What's wrong with the answer in the link you added? This is the way to go – Simon Mourier Apr 22 '20 at 15:43
  • @SimonMourier look at the comments of the post, this approach helps me recieve only the x,y coordinates, which is, as I mentioned, not what I need. – Tal XD Apr 22 '20 at 17:03
  • If the textbox is a standard Windows textbox, then you can send the textbox an [`EM_CHARFROMPOS`](https://learn.microsoft.com/en-us/windows/win32/controls/em-charfrompos) window message to convert client coordinates to a character index. Or, send it an [`EM_GETSEL`](https://learn.microsoft.com/en-us/windows/win32/controls/em-getsel) window message to get the start/end indexes of the currently selected text, one of which will be the same as the caret position. – Remy Lebeau Apr 22 '20 at 19:49
  • EM_CHARFROMPOS always return 0 @RemyLebeau – Tal XD Apr 26 '20 at 13:26
  • @TalXD did you convert your coordinates from screen coordinates to client coordinated first? – Remy Lebeau Apr 26 '20 at 19:10
  • @RemyLebeau I didn't know that I need to, I don't know what it means. – Tal XD Apr 27 '20 at 06:13
  • @TalXD there are two kinds of coordinates, coordinates within the overall [virtual screen](https://learn.microsoft.com/en-us/windows/win32/gdi/the-virtual-screen), and coordinates within the client area of each HWND. The other solution you linked to says it provides the caret position in screen coordinates. `EM_CHARFROMPOS` takes client coordinates. You can use `ScreenToClient()` or `MapWindowPoints()` to convert screen coordinates into client coordinates. – Remy Lebeau Apr 27 '20 at 15:01
  • Thanks. and how do I use EM_getsel? How do I cast the int value to the start and end index? @RemyLebeau – Tal XD Apr 28 '20 at 10:11

1 Answers1

2

You can use UI Automation for that, and especially the IUIAutomationTextPattern2 interface that has a GetCaretRange method.

Here are two sample Console app (C++ and C# code) that run continuously and display the caret position for the current element under the mouse:

C++ version

int main()
{
    CoInitializeEx(NULL, COINIT_MULTITHREADED);
    {
        CComPtr<IUIAutomation> automation;

        // make sure you use CLSID_CUIAutomation8, *not* CLSID_CUIAutomation
        automation.CoCreateInstance(CLSID_CUIAutomation8);
        do
        {
            POINT pt;
            if (GetCursorPos(&pt))
            {
                CComPtr<IUIAutomationElement> element;
                automation->ElementFromPoint(pt, &element);
                if (element)
                {
                    CComBSTR name;
                    element->get_CurrentName(&name);
                    wprintf(L"Watched element %s\n", name);

                    CComPtr<IUIAutomationTextPattern2> text;
                    element->GetCurrentPatternAs(UIA_TextPattern2Id, IID_PPV_ARGS(&text));
                    if (text)
                    {
                        // get document range
                        CComPtr<IUIAutomationTextRange> documentRange;
                        text->get_DocumentRange(&documentRange);

                        // get caret range
                        BOOL active = FALSE;
                        CComPtr<IUIAutomationTextRange> range;
                        text->GetCaretRange(&active, &range);
                        if (range)
                        {
                            // compare caret start with document start
                            int caretPos = 0;
                            range->CompareEndpoints(TextPatternRangeEndpoint_Start, documentRange, TextPatternRangeEndpoint_Start, &caretPos);
                            wprintf(L" caret is at %i\n", caretPos);
                        }
                    }
                }
            }
            Sleep(500);
        } while (TRUE);
    }
    CoUninitialize();
    return 0;
}

C# version

static void Main(string[] args)
{
    // needs 'using UIAutomationClient;'
    // to reference UIA, don't use the .NET assembly
    // but instead, reference the UIAutomationClient dll as a COM object
    // and set Embed Interop Types to False for the UIAutomationClient reference in the C# project
    var automation = new CUIAutomation8();
    do
    {
        var cursor = System.Windows.Forms.Cursor.Position;
        var element = automation.ElementFromPoint(new tagPOINT { x = cursor.X, y = cursor.Y });
        if (element != null)
        {
            Console.WriteLine("Watched element " + element.CurrentName);
            var guid = typeof(IUIAutomationTextPattern2).GUID;
            var ptr = element.GetCurrentPatternAs(UIA_PatternIds.UIA_TextPattern2Id, ref guid);
            if (ptr != IntPtr.Zero)
            {
                var pattern = (IUIAutomationTextPattern2)Marshal.GetObjectForIUnknown(ptr);
                if (pattern != null)
                {
                    var documentRange = pattern.DocumentRange;
                    var caretRange = pattern.GetCaretRange(out _);
                    if (caretRange != null)
                    {
                        var caretPos = caretRange.CompareEndpoints(
                            TextPatternRangeEndpoint.TextPatternRangeEndpoint_Start,
                            documentRange,
                            TextPatternRangeEndpoint.TextPatternRangeEndpoint_Start);
                        Console.WriteLine(" caret is at " + caretPos);
                    }
                }
            }
        }
        Thread.Sleep(500);
    }
    while (true);
}

The trick is to use the IUIAutomationTextRange::CompareEndpoints method that allows you to compare the caret range with another range, for example the whole document range.

Note there are drawbacks:

  • some apps don't support any MSAA/UIA introspection at all, or don't support the text pattern. For these, there are simply no solution (even using Windows API I think)
  • some apps report the caret incorrectly, especially when you select text (so, with a moving caret). For example with Notepad, moving LEFT while pressing SHIFT will select backwards but for some reason UIA doesn't update caret pos. I think it's a problem with Notepad because it also has caret issue with IME (Input Method Editor, like the Emoji editor you can summon using Win+; keyboard combination) which also uses the global caret position. There's no problem with Wordpad for example.
Simon Mourier
  • 132,049
  • 21
  • 248
  • 298
  • Can I do the same thing in c#? Also Thanks! @SimonMourier – Tal XD Apr 23 '20 at 05:46
  • I have copied yor code, and for some reason vs doesn't find classes like CUIAutomation8 and IUIAutomationTextPattern2 – Tal XD Apr 26 '20 at 12:42
  • @TalXD - have you referenced UIAutomationClient dll as a COM object, not using standard .NET assemblies? What Windows version are you using? – Simon Mourier Apr 26 '20 at 13:52
  • win 10, and about an hour ago I did and tried your code. But for some reason it doesn't work on google chrome, it probably doesn't support AutomationUI. Thanks!!! – Tal XD Apr 26 '20 at 17:37
  • 1
    @TalXD - chrome can be extensively configured for automation in general: https://stackoverflow.com/questions/47216824/does-microsoft-ui-automation-framework-work-with-chrome-python-and-java-apps – Simon Mourier Apr 26 '20 at 18:59
  • @SimonMouirer thanks a lot! but it is still not what I need, I'm trying to know what is the caret position whenever chrome is opened. But maybe I can make chrome open in this mode whenever it's opened. Is this possible? – Tal XD Apr 27 '20 at 06:16
  • @TalXD - for any given installation, this is not possible AFAIK. You can configure a special shortcut for accessibility, or detect accessibility is not there and tell the user – Simon Mourier Apr 27 '20 at 06:57