Finding _and then clicking_ ListBox Items from another program built in VB6 (ThunderRT6ListBox)

Question

wracking my brain on this one.

I have successfully utilized this link to successfully grab text of the items of a ListBox (ThunderRT6ListBox specifically) working with WinAPI functions. What I want to accomplish is to select and then double-click the necessary item from this list. I've successfully been able to find the WindowRect of the ListBox itself, but I cannot get the cursor position relative to this Box and desired item.

I have to submit a double-click on this as the author of the application I'm ScreenScraping set the double-click event of the item to open a child window where parameters are set relative to the selected item.

I'm only going to post the part where I'm having trouble - again - I've successfully found the data and have been able to get the WindowRect of the ListBox - I'm now trying to click and double-click the item needed. I imagine I need to then set my mouse cursor position using another winAPI, but when attempting to get the WindowRect of the Item it fails and gives a 0 for both x,y coordinates. I have also tried SendKeys and other variants of this with an attempt to "Select" the item by value. . . this also fails.

I am writing in VB.NET but do not mind answers posted with C#, as there are converters and these answers are more prevalent for everything anyway (which is also helping me to learn that language too).

 Public Function GetListBoxContents(ByVal listBoxHwnd As IntPtr) As List(Of String)
    Dim cnt As Integer = CInt(SendMessage(listBoxHwnd, LB_GETCOUNT, IntPtr.Zero, Nothing))

    Dim listBoxContent As List(Of String) = New List(Of String)()

    For i As Integer = 0 To cnt - 1
      Dim sb As StringBuilder = New StringBuilder(256)
      Dim getTextLen As Integer = CInt(SendMessage(listBoxHwnd, LB_GETTEXTLEN, Nothing, Nothing))
      Dim getText As IntPtr = SendMessage(listBoxHwnd, LB_GETTEXT, CType(i, IntPtr), sb)
      Dim ComRect As RECT
      GetWindowRect(listBoxHwnd, ComRect)
      Debug.Print("Rect's x coordinate = " & ComRect.x) 'Produces valid X coord when looking
                                                         at listboxHwnd - if I look at "getText"
                                                         it fails - I assume because this really
                                                         is not 
      Debug.Print("Rect's y coordinate = " & ComRect.y)
      Dim lparam As Integer = MakeLong(ComRect.x, ComRect.y)
      SetForegroundWindow(listBoxHwnd)
      If sb.ToString Like "*COM5*" Then
        Debug.Print("hWnd of ListBox item = ", getText)
        WindowsEnumerator.SendMessage(getText, WM_LBUTTONDOWN, &H1, lparam)
        WindowsEnumerator.SendMessage(getText, WM_LBUTTONUP, &H1, lparam)

        PostMessage(getText, WM_LBUTTONDBLCLK, &H1, lparam)

        PostMessage(getText, BM_CLICK, &H1, lparam)

        SendInputs.Mouse_Click()

        SendInputs.SendKey(Convert.ToChar(13))

        SendKeys.SendWait("{ENTER}")

      End If
      listBoxContent.Add(sb.ToString)
    Next

    Return listBoxContent
  End Function

Answer 1

After losing some of my hair, I finally managed to figure out how to do what I need.

HUGE kudos to Unhnd_Exception on the Daniweb site for this neat, intuitive class. (I'm linking to the post, but also posting the content of Unhnd_Exception's class here in case that site ever goes down. If I have not attributed it enough to the user, please help me make a better approach to this without losing the code - this was a life-saver! (REF#1))

Also - I think ANYONE who dabbles in the API Realm who has struggled to find the ACTUAL values of the Constants will really appreciate this following link. It saved my bacon on the remainder of the code. (REF#2) Additionally, having the MSDN Page for List Box Messages was the other Key I needed as it contained the different methods to communicate with these objects! (REF#3)

Final Methodology used to accomplish my task:

  Public Const LB_GETCOUNT As Integer = &H18B ' Gets the count of items in ListBox
  Public Const LB_GETTEXTLEN As Integer = &H18A ' Gets Length of Text for given
                                                  Item based on hWnd
  Public Const LB_GETTEXT As Integer = &H189 ' Gets Text
  Public Const LB_GETITEMRECT As Integer = &H198 ' Gets ListBox Item RECT - NOTE:
                                                 ' This still failed for me and returned
                                                 ' "0" for all dimensions in this case!
  Public Const LB_SELECTSTRING As Integer = &H18C ' Hoped this was useful - not so much
  Public Const LB_SETCURSEL As Integer = &H186 ' THIS ONE - NICE Beauty!
              'LB_SETCURSEL will focus the selection on the object by index
              'It will also scroll the box if there are enough items.
  Public Const LB_SETSEL As Integer = &H185 ' Ultimately did not use this one.


  Public Function GetListBoxContents(ByVal listBoxHwnd As IntPtr) As List(Of String)
    Dim cnt As Integer = CInt(SendMessage(listBoxHwnd, LB_GETCOUNT, IntPtr.Zero, Nothing))

    Dim listBoxContent As List(Of String) = New List(Of String)()

    For i As Integer = 0 To cnt - 1
      Dim sb As StringBuilder = New StringBuilder(256)
      Dim itmRect As RECT
      Dim getTextLen As Integer = CInt(SendMessage(listBoxHwnd, LB_GETTEXTLEN, Nothing, Nothing))
      Dim getText As IntPtr = SendMessage(listBoxHwnd, LB_GETTEXT, CType(i, IntPtr), sb)

      SendMessage(getText, LB_SETCURSEL, CType(i, IntPtr), Nothing)
      Dim ComRect As RECT
      GetWindowRect(listBoxHwnd, ComRect)
      Debug.Print("Rect's x coordinate = " & ComRect.x)
      Debug.Print("Rect's y coordinate = " & ComRect.y)
      Debug.Print("Rect's h coordinate = " & ComRect.h)
      Debug.Print("Rect's w coordinate = " & ComRect.w)
      SendMessageRect(getText, LB_GETITEMRECT, CType(i, IntPtr), itmRect)
      Debug.Print("itmRect's x coordinate = " & itmRect.x) 'All of these returned 0's
      Debug.Print("itmRect's y coordinate = " & itmRect.y) 'Perhaps I'm calling the routine
      Debug.Print("itmRect's h coordinate = " & itmRect.h) 'incorrectly???  Would love to know
      Debug.Print("itmRect's w coordinate = " & itmRect.w) 'why this fails.
      SetCursorPos(ComRect.x + 10, ComRect.h - 10) 'THIS ONE.  Moves my cursor in position
                                                   'to select the appropriate object.
                                                   'If I could make the GETITEMRECT
      SetForegroundWindow(listBoxHwnd)             'Method to work, this would be MOST ideal
                                                   'As it would always be concurrent
                                                   'with the given item and be less
      If sb.ToString Like "*COM5*" Then            'prone to failure. Moves selection to bottom.
        Debug.Print("hWnd of ListBox item = ", getText)
        SendMessage(listBoxHwnd, LB_SETCURSEL, CType(i, IntPtr), Nothing)
        SendInputs.Double_Click() ' Here is the clicker - worked like a champ

      End If
      listBoxContent.Add(sb.ToString)
    Next
    Return listBoxContent
  End Function

Sendinput that actually works (REF#1)

Value of the constants for the Windows 32-bit API (REF#2)

List Box Messages (REF#3)

Code used from Unhnd_Exception on Daniweb in entirety:

Imports System.Runtime.InteropServices

Public Class SendInputs

    Private Const KeyDown As Integer = &H0
    Private Const KeyUp As Integer = &H2

    <DllImport("user32.dll")> _
  Private Shared Function SendInput( _
        ByVal nInputs As Integer, _
        ByVal pInputs() As INPUT, _
        ByVal cbSize As Integer) As Integer
    End Function

    <StructLayout(LayoutKind.Explicit)> _
   Private Structure INPUT
        'Field offset 32 bit machine 4
        '64 bit machine 8
        <FieldOffset(0)> _
        Public type As Integer
        <FieldOffset(8)> _
        Public mi As MOUSEINPUT
        <FieldOffset(8)> _
        Public ki As KEYBDINPUT
        <FieldOffset(8)> _
        Public hi As HARDWAREINPUT
    End Structure

    Private Structure MOUSEINPUT
        Public dx As Integer
        Public dy As Integer
        Public mouseData As Integer
        Public dwFlags As Integer
        Public time As Integer
        Public dwExtraInfo As IntPtr
    End Structure

    Private Structure KEYBDINPUT
        Public wVk As Short
        Public wScan As Short
        Public dwFlags As Integer
        Public time As Integer
        Public dwExtraInfo As IntPtr
    End Structure

    Private Structure HARDWAREINPUT
        Public uMsg As Integer
        Public wParamL As Short
        Public wParamH As Short
    End Structure

      Public Shared Sub SendKey(ByVal key As Char)
        Dim Inpts(1) As INPUT

        'key down
        Inpts(0).type = 1
        Inpts(0).ki.wVk = Convert.ToInt16(CChar(key))
        Inpts(0).ki.dwFlags = KeyDown

        'key up
        Inpts(1).type = 1
        Inpts(1).ki.wVk = Convert.ToInt16(CChar(key))
        Inpts(1).ki.dwFlags = KeyUp

        SendInput(2, Inpts, Marshal.SizeOf(GetType(INPUT)))
    End Sub
End Class

I modified his Button click Subroutine (full content shown below) to accomplish what I need (removed the binding to a button click within the program). If you duplicate the final line SendInput(2, Inpts, Marshal.SizeOf(GetType(INPUT))) you essentially get 2 clicks in rapid succession. In my implementation I added a "Mouse_Click" routine (as mentioned - no WinForm Control needed), and I also implemented a "Double_Click" routine which gives the result I need.)

Private Sub Button3_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button3.Click
        'raises the key down and up events

        Dim Inpts(1) As INPUT

        'key down
        Inpts(0).type = 1
        Inpts(0).ki.wVk = Convert.ToInt16(CChar("J"))
        Inpts(0).ki.dwFlags = 0

        'key up
        Inpts(1).type = 1
        Inpts(1).ki.wVk = Convert.ToInt16(CChar("J"))
        Inpts(1).ki.dwFlags = 2

        SendInput(2, Inpts, Marshal.SizeOf(GetType(INPUT)))
    End Sub

Answer 2

Based on the suggestions from both @IInspectable and @Charles, UI Automation is DEFINITELY the way to go. I've always had a bit of a problem trudging through the MSDN Articles for each method because often times they use C++ or C# code (I guess someday I'll need to really learn these languages, but I digress).

For those who are intimidated by the UI Automation Approach, I'm leaving my other answer because it would be useful if you can master it. . . but it is literally reinventing UI Automation to try and DO IT!

Using the MSDN Documentation mostly confused me until I learned the structures of the Elements/Properties, etc. (Further research also shows I could combine properties using AndCondition to build conjunctive properties. This is well and great, however, I have to credit This YouTube Video which takes a UI Automation approach to reading text for getting me over the hump. Sure it's a totally different approach/use case, BUT the structures are the same, and the author of the Video (Tech Savvy) uses VB.Net language which is what I really needed.

Below is the entire subroutine I built - from scratch yet! I'm sure I can improve it, but for a first attempt at working with UI Automation I was able to accomplish the ENTIRE process needed, NOT just only a smidgen of some of the work, and it is MUCH. . . . MUCH easier long-run, so THANK YOU ALL for the suggestions to move to UI Automation - this is ABSOLUTELY the best move! I felt like I gained a 'Level Up' in my coding. <3

Note: When dealing with legacy objects from VB6 (Thunder Frames, etc), there is thankfully a LocalizedControlType Property which can be used to find your items as you walk the subtree of your initial program!

Note2: You'll notice I'm using a Click_Object routine for my calls. . . This is NOT using the Invoke method because I had trouble with Invoke calls hanging for up to 10 Seconds or more for a simple button click. I know I can call these using a separate thread dispatcher, but I ultimately went with Microsft.TestAPI's Mouse Class and the Click/MoveTo Methods therein as suggested in This SO Article.

  Public Shared Sub FindHMUtil(ByVal ComNum As String)
    Dim mW As MainWindow = CType(Application.Current.MainWindow, MainWindow)
    'If utility is open - need to kill it and restart.
    If Process.GetProcessesByName("hmutility").Count > 0 Then
      Commands(0, 0) = "/C TaskKill /FI ""imagename eq hmutility.exe"" /F"
      Commands(1, 0) = "Successfully able to kill Firmware Utility."
      Commands(2, 0) = "Unable TypeOf kill Firmware Utility."
      Commands(3, 0) = "Restarting TPG Printer Firmware Utility"
      Commands(4, 0) = "Confirming TPG Printer Firmware"
      Commands(5, 0) = "False"
      MyProcessControl.ProcessCommandLine("cmd.exe", Commands, mW.PrinterCheckText2)

    End If

    'Restart Utility
    Commands(0, 0) = "/C Start ""HmUtil"" ""C:\Program Files (x86)\hmUtility\hmUtility.exe"""
    Commands(1, 0) = "Successfully Able to Start Firmware Utility"
    Commands(2, 0) = "Unable to Start Firmware Utility"
    Commands(3, 0) = "Restarting TPG Printer Firmware Utility"
    Commands(4, 0) = "Confirming TPG Printer Firmware"
    Commands(5, 0) = "False"
    MyProcessControl.ProcessCommandLine("cmd.exe", Commands, mW.PrinterCheckText2)
    Thread.Sleep(350)

    Dim enumerator As New WindowsEnumerator

    'get handle for initial window
    'Note - FindWindow API will not find ThunderForm Windows with WindowTitle if they are part of a Thunder Wrapper!!!!
    'Leaving this as a construct for myself
    Dim hmImg As IntPtr = enumerator.FindWindowInt(Nothing, "Image")
    If hmImg = IntPtr.Zero Or hmImg = vbEmpty Or hmImg = 0 Or hmImg = vbNull Then
      'keeping as a reminder - sometimes the findwindow API will not find the ThunderForm
      hmImg = enumerator.FindWindowInt(ThunderForm, Nothing)
      If hmImg = IntPtr.Zero Or hmImg = vbEmpty Or hmImg = 0 Or hmImg = vbNull Then
        hmImg = enumerator.FindWindowExInt(Nothing, Nothing, ThunderForm, Nothing)
        Debug.Print("Still found nothing mate!")
      End If
    End If
    'get initial window element
    Dim hmWind As AutomationElement = AutomationElement.FromHandle(hmImg)

    'Get first set of Elements needed from initial window.  NOTE: Not all are used immediately, but keeping them read prevents need to find them a 2nd time.
    Dim hmWindExecute As AutomationElement = hmWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Execute"))
    Dim hmWindResetPrntr As AutomationElement = hmWindExecute.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Reset Printer"))
    Dim hmWindSetup As AutomationElement = hmWindExecute.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Setup"))
    Dim hmWindSend As AutomationElement = hmWindExecute.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Send"))
    Dim hmWindClearStat As AutomationElement = hmWindExecute.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Clear Status"))
    Dim hmWindFLashHndl As IntPtr = enumerator.FindWindowExInt(Nothing, Nothing, ThunderFrame, "Flash File Selection")
    Dim hmWindPane As AutomationElement = hmWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.LocalizedControlTypeProperty, "pane"))
    Dim hmWindPaneBtns As AutomationElementCollection = hmWindPane.FindAll(TreeScope.Subtree, New PropertyCondition(AutomationElement.LocalizedControlTypeProperty, "button"))

    Click_Object(hmWindPaneBtns(4), hmWindPaneBtns(4).GetClickablePoint)

    Dim hmWindFlashFileFrame As AutomationElement = hmWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Flash File Selection"))
    Dim hmWindFlashFileBox As AutomationElement = hmWindFlashFileFrame.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.LocalizedControlTypeProperty, "edit"))

    Click_Object(hmWindSetup, hmWindSetup.GetClickablePoint)

    'Get handle for new setup window
    Dim hmSetup As IntPtr = enumerator.FindWindowExInt(Nothing, Nothing, ThunderForm, "Setup")
    If hmSetup = IntPtr.Zero Or hmSetup = 0 Or hmSetup = vbEmpty Or hmSetup = vbNull Then
      hmSetup = enumerator.FindWindowExInt(Nothing, Nothing, ThunderForm, Nothing)
    End If

    'Get new window with listbox of ComPorts and get other controls inside new window
    Dim hmSetupWind As AutomationElement = AutomationElement.FromHandle(hmSetup)
    Dim hMSetupRemove As AutomationElement = hmSetupWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Remove"))
    Dim hmSetupOk As AutomationElement = hmSetupWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "OK"))
    Dim hmSetupON As AutomationElement = hmSetupWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "On"))
    Dim hmSetupClear As AutomationElement = hmSetupWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Clear"))
    Dim hmSetupStatList As AutomationElement
    Dim hmListHndl As IntPtr = enumerator.FindWindowExInt(Nothing, Nothing, ThunderList, Nothing)


    Dim hmListBox As AutomationElement
    If hmListHndl = IntPtr.Zero Or hmListHndl = 0 Or hmListHndl = vbEmpty Or hmListHndl = vbNull Then
      hmListBox = hmSetupWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.LocalizedControlTypeProperty, "list"))
    Else
      hmListBox = AutomationElement.FromHandle(hmListHndl)

    End If
    'Get all ListBoxes of the new window
    Thread.Sleep(250)


    Dim hmSetupParent As TreeWalker
    'iterating through all listboxes - if not original list box then
    Dim checkItm As AutomationElement
    Dim checkItmCache As New CacheRequest
    checkItmCache.Add(AutomationElement.LocalizedControlTypeProperty)
    Click_Object(hMSetupRemove, hMSetupRemove.GetClickablePoint)

    Dim hmListItems As AutomationElementCollection = hmListBox.FindAll(TreeScope.Subtree, New PropertyCondition(AutomationElement.LocalizedControlTypeProperty, "list item"))



    For Each hmListItem As AutomationElement In hmListItems
      If hmListItem.Current.Name = ComNum Then
        'DoubleClick list item for ComPort of Printer
        hmListItem.SetFocus()
        DoubleClick_Object(hmListItem, hmListItem.GetClickablePoint)
        'Get info for new Window
        Dim hmPortSetupWindHndl As IntPtr = enumerator.FindWindowExInt(Nothing, Nothing, ThunderForm, Nothing)
        Dim hmPortsetupOK As AutomationElement = AutomationElement.FromHandle(hmPortSetupWindHndl).FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Ok"))
        'Click OK on new window with Port Settings - Defaults are fine.
        Click_Object(hmPortsetupOK, hmPortsetupOK.GetClickablePoint)

        'Turn port ON
        Click_Object(hmSetupON, hmSetupON.GetClickablePoint)

        'Get items from Status Boxt
        Dim hmSetupStatFind As TreeWalker

        Click_Object(hmSetupOk, hmSetupOk.GetClickablePoint)
        Exit For
      End If
    Next hmListItem

    Click_Object(hmWindFlashFileBox, hmWindFlashFileBox.GetClickablePoint)
    My.Computer.Clipboard.SetText(MyProcessControl.myPath & "TPG\189-798F304A.BIN")
    My.Computer.Keyboard.SendKeys("^V", True)

    Click_Object(hmWindSend, hmWindSend.GetClickablePoint)

    Dim hmStatus As AutomationElement = hmWind.FindFirst(TreeScope.Subtree, New PropertyCondition(AutomationElement.NameProperty, "Status"))
    Dim SectorStrng As String = "Derps"
    Do Until SectorStrng.Contains(ComNum & " open")
      Dim hmStatLines As AutomationElementCollection = hmStatus.FindAll(TreeScope.Subtree, New PropertyCondition(AutomationElement.LocalizedControlTypeProperty, "list item"))
      For a = 0 To hmStatLines.Count - 1
        If hmStatLines(a).Current.Name.Contains(ComNum & " open") Then
          SectorStrng = hmStatLines(a).Current.Name
        End If
      Next
      Dim e As EventArgs
      MyProcessControl.ForceUIToUpdate()
      MyProcessControl.OnTimedEvent(mW, e)
    Loop


    If Process.GetProcessesByName("hmutility").Count > 0 Then
      Commands(0, 0) = "/C TaskKill /FI ""imagename eq hmutility.exe"" /F"
      Commands(1, 0) = "Successfully able to kill Firmware Utility."
      Commands(2, 0) = "Unable TypeOf kill Firmware Utility."
      Commands(3, 0) = "Restarting TPG Printer Firmware Utility"
      Commands(4, 0) = "Confirming TPG Printer Firmware"
      Commands(5, 0) = "False"
      MyProcessControl.ProcessCommandLine("cmd.exe", Commands, mW.PrinterCheckText2)

    End If

  End Sub

  Public Shared Sub DoubleClick_Object(givenElement As AutomationElement, mousePoint As Point)
    givenElement.SetFocus()
    'Adding a 1/4 second sleep ensures object has proper focus before clicking - sometimes the routines moved too fast and threw an exception.
    Thread.Sleep(250)
    Input.Mouse.MoveTo(New System.Drawing.Point(mousePoint.X, mousePoint.Y))
    Input.Mouse.DoubleClick(Input.MouseButton.Left)
  End Sub

  Public Shared Sub Click_Object(givenElement As AutomationElement, mousePoint As Point)

    givenElement.SetFocus()
    Thread.Sleep(250)
    Input.Mouse.MoveTo(New System.Drawing.Point(mousePoint.X, mousePoint.Y))
    Input.Mouse.Click(Input.MouseButton.Left)

  End Sub