1

I want to record a user scenario of a windows application. For this I want to retrieve/log the object type, the object name and the object value of GUI objects after a mouse click on the object next to a screen capture and the mouse position.

I am stucked with finding the how to's. Please help.

I have the following Python code. 

saveDir = r'C:\Privat\ZuppU\temp'
stepListID = 1


def capture_step_objects(event):
    global stepListID
    stepEvent = 'Click'

    # Capture the application mouseclick area
    (mx, my) = event.Position

    fgWindow = win32gui.GetForegroundWindow()
    fgWindowName = win32gui.GetWindowText(fgWindow)

    # Capture the application window
    (ax1, ay1, ax2, ay2) = win32gui.GetWindowRect(fgWindow)
    stepCapture = ImageGrab.grab((ax1, ay1, ax2, ay2))
    stepImage = os.path.join(saveDir, 'Step' + str(stepListID) + '.png')
    stepCapture.save(stepImage)

hm = pyHook.HookManager()
hm.SubscribeMouseAllButtonsDown(capture_step_objects)
hm.HookMouse()
pythoncom.PumpMessages()
hm.UnhookMouse()

# Main loop
demonplus
  • 5,613
  • 12
  • 49
  • 68
Bill Bridge
  • 821
  • 1
  • 9
  • 30

1 Answers1

0

There is an example for pywinauto and pyHook: recorder.py.

But you may stuck with some other pitfalls. The typical example is closing the dialog by button "OK" click. When you're in the event handler, the dialog may be already closed so you will not be able to retrieve any info about non-existing window. Some complicated tracking is required here.

The same thing relates to UI Automation API events (pywinauto is going to support UIA soon). But we have no resources to implement even pure Python analogue of pyHook in a short term. So a good recorder implementation looks far far away from now. Any help is appreciated. ;)

Vasily Ryabov
  • 9,386
  • 6
  • 25
  • 78