Retrieve the element name and xpath when clicking a button or typing in a textbox

Question

I want to develop an app that helps me to record all the keyboard and mouse events executed in a particular Windows Application for example in Microsoft Excel, Acrobat, Notepad, and so on.

I tried Pyhook and Win32gui to achieve my goal so far. However, I do not know how to retrieve the following information:

Target element name which was clicked or where user wrote something (For example if the application is notepad I would like to store "Format" when the user clicked on Format menu)
Xpath of that target element name (Following the example above, I want to know, for that menu Format, its parent handle and respective class parent)

Thank you so much for your help or advice and please forgive me if I wrote something incorrectly. I am very new with Python ;)

Drake Wu · Accepted Answer · 2020-01-22T03:42:54.070

First get the current mouse click position, use WindowFromPoint to get the current window handle;

GetClassName to get the window class name;

GetMenu to get the HMENU handle;

Then use MenuItemFromPoint to get the menu item ID(this function return will -1 if no menu item is at the specified location);

Finally use GetMenuItemInfo(The GetMenuString function has been superseded by GetMenuItemInfo) to get the menu text.

Here is a simple C++ test sample:

#include <Windows.h>
#include <iostream>

int main()
{
    Sleep(2000);//give us the time to test clicking on the menu.
    POINT p;
    GetCursorPos(&p);
    HWND h = WindowFromPoint(p);
    char classname[100] = { 0 };
    GetClassName(h, classname,100);

    MENUBARINFO menubar = {0};
    menubar.cbSize = sizeof(MENUBARINFO);
    HMENU menu = GetMenu(h);
    int id = MenuItemFromPoint(h, menu,p);

    MENUITEMINFO info = { 0 };
    info.cbSize = sizeof(MENUITEMINFO);
    info.fMask = MIIM_STRING;
    info.dwTypeData = NULL;

    GetMenuItemInfo(menu, id , true,&info);
    info.dwTypeData = (LPSTR)malloc(info.cch+1);
    info.cch++;
    GetMenuItemInfo(menu, id, true, &info);

    MessageBox(NULL, info.dwTypeData,"Menu Name",0);
    free(info.dwTypeData);
    return 0;
}

UPDATE:

That's the code I have tested, and work for me.(test in NotePad)

import win32api
import win32gui
import win32gui_struct
import win32con
import time
import ctypes
from ctypes.wintypes import tagPOINT

time.sleep(3) #point to the menu before the time ends.
pos = win32gui.GetCursorPos()
hwnd = win32gui.WindowFromPoint(pos)

##get ClassName
ClassName = win32gui.GetClassName(hwnd)
menu = win32gui.GetMenu(hwnd)
print("ClassName = " + ClassName)

##get Id
point = tagPOINT(pos[0],pos[1])
Id = ctypes.windll.user32.MenuItemFromPoint(hwnd,menu,point)
print("Id = " + str(Id))

##get Menu
info,extras = win32gui_struct.EmptyMENUITEMINFO(win32con.MIIM_STRING)
win32gui.GetMenuItemInfo(menu,Id,1,info)
strings = win32gui_struct.UnpackMENUITEMINFO(info)
print("Menu = " + strings.text)

Hi @Drake Wu, thank you so much for your help. I am converting this code into Python and I got stuck when I want to define MENUITEMINFO and to pass this parameter into GetMenuItemInfo. After a long search, I found this way to create this object: info, extras = =win32gui_struct.EmptyMENUITEMINFO(win32con.MIIM_STRING). However, when I want to pass this object into the function GetMenuItemInfo, I got an error that says that It is a incorrect parameter. Do you know how should I implement it in Python?. Thanks again for your help. — sergioMoreno, Dec 23 '19 at 13:16
Hi @Drake Wu, thanks again for your valuable help and patience. I would like to confirm with you if I am getting the ID correctly because I got the same error. This is what I did to get the ID: x,y = win32gui.GetCursorPos(), point = tagPOINT(x,y) , Id = ctypes.windll.user32.MenuItemFromPoint(hwnd,menu,point). It is correct? — sergioMoreno, Jan 21 '20 at 10:49
Hi @Drake Wu tks for your help. It works perfectly with Note pad. I learned a lot with your help, tks again for your time helping me out. I was trying the same code with other windows applications but unfortunately the menu was 0 or Id was -1. Do you know how to identify the name of a button when it is not a menu itself?. Tks anywar for your support. — sergioMoreno, Jan 22 '20 at 13:24
Try to use [UI Automation](https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32) — Drake Wu, Jan 23 '20 at 01:16
With [`ElementFromPoint`](https://learn.microsoft.com/en-us/windows/win32/api/uiautomationclient/nf-uiautomationclient-iuiautomation-elementfrompoint) and [`get_CurrentName`](https://learn.microsoft.com/en-us/windows/win32/api/uiautomationclient/nf-uiautomationclient-iuiautomationelement-get_currentname). — Drake Wu, Jan 23 '20 at 01:22
Tks again @Drake Wu.... I will investigate UI Automation. Tks for sharing your knowledge and information. All the best — sergioMoreno, Jan 23 '20 at 13:54

Retrieve the element name and xpath when clicking a button or typing in a textbox

1 Answers1