I am beyond tired researching this topic at this point. There are a million APIs in the system to achieve part of this, i.e. enumerate all windows, but nothing about how to get them in the real z-order. How to do that? Let me explain.
Firstly, of course, to enumerate windows, one uses EnumWindows or when that behaves poorly and decides not to show UWP apps because of some stupid limitation in the OS one can directly use the NtUserBuildHwndList which is anyway called in the backend by EnumWindows.
With that out of the way, in the callback you can filter child, hidden and weird windows. Depending on what you do, you will get varying results. To spare you the read, the most common approaches are Raymond Chen's kind of outdated solution or try something more up to date which better filters hidden windows in Windows 10+, and also augment it with DwmGetWindowAttribute(hwnd, DWMWA_CLOAKED, &bIsCloaked, sizeof(BOOL)); with BOOL bIsCloaked = TRUE;, you eventually get the real windows that you are interested in on the desktop.
The only problem is, they are kind of sorted in the z-order. The list is made by sorting all windows in each window band by z-order, and then the results are concatenated in a final list. Now, what is a window band? It is an undocumented Microsoft invention, in which they made some of their windows sit even more on top than HWND_TOPMOST (makes sense from a security point of view, it does not make sense in the sense of not documenting it at all). You can read about this here. Basically, there are multiple bands on the desktop, each band being a group of windows that share a z-order. Then, bands are stacked one on top of the other, generating a z-order for bands, let's say. Task Manager with 'Always on top' checked runs in the ZBID_SYSTEM_TOOLS band, which is above ZBID_DESKTOP, that's why it always is on top of any window a regular app can create.
This makes perfect sense, but the user does not care about window bands. It sees windows on the desktop, and when Alt-Tabing, it expects a list in descending order from most recently used to the last window to have been used. In this order, despite of the band. So, for example, my workflow was Firefox-Task Manager-cmd, I expect to see cmd-Task Manager-Firefox in Alt-Tab, despite Task Manager having always on top.
It is clear by now that one needs to somehow sort this list. The question that arises is, how do you sort a window list by z-order disregarding the window band? I cannot for the life in me find anything useful anywhere, neither on Google, nor in the lackluster documentation. GetTopWindow and GetNextWindow, the traditional functions let's say, always put the first the windows from other bands than ZBID_DESKTOP, and then properly sort the rest of the windows in ZBID_DESKTOP. All legacy APIs just behave like windows in other bands are HWND_TOPMOST, which make sense. But I don't want that: z-order = activation order only when we consider all the bands together or when all the windows are on the same band. I want windows in activation order, which is close to z-order, but not the same, as I have tried to explain here.
Obviously Windows does it somehow, as the behavior I describe is taking place in Alt-Tab, Win-Tab, Snap Assist and so on. There clearly is an API (that does not require administrator, as Explorer runs as regular user) that allows you to sort windows in such an order, or get a sorted list directly. No, I don't think Explorer tracks the order internally, as when you restart it and immediately hit Alt-Tab or something like that, the window appear in correct order, or so it seems to me. The window were already created, so it must grab the info from somewhere.
I also looked in twinui.pcshell.dll which is basically the desktop Windows shell at this point, but haven't been able to find anything relevant as of yet. There are some methods with zorder in their name, but of course a million dynamic dispatch calls happen, and you get lost very easily when dynamically walking through the call stack, and for the life in me I cannot find the comparator they use when sorting them (for example, AppViewSwitchItemZOrderSort::AppViewSwitchItemZOrderSort gets called when hitting Win+Tab).
Does anyone have any pointers in this direction, I would greatly appreciate it. I prefer C, but anything at this point is serviceable, I just want to know what the API/COM call/whatever to do is.
P.S. Everything these days in Windows is reverse engineering from the ground up. Microsoft has made an entire different set of APIs just for their use, and left everyone else with rusting old APIs just so that their built-in functions perform better. Their Alt+Tab or Win+Tab has live window previous. You'd think they use the DWM thumbnails API, but, of course no, they use some internal API that allows them to manipulate the texture or get it with rounded corners, so it looks proper on Windows 11, Of course, they also haven't bothered to update the DWM thumbnails API to have apps be able to request textures with rounded corners. Because why bother. They don't even bother with their own software anymore. Just compile some code that opens a window and closes it in 10 seconds. In those 10 seconds, open their Alt-Tab in Windows 11 and just sit there. When the window closes, Explorer crashes, because no one expected a window to close on its own... like, c'mon Microsoft...
Thank you.
