0

I've been working for days on an assembler subroutine to call Windows API function FormatMessageA, and I think I must have some systematic misunderstanding. My routine is shown below. I have traced it in debug mode, and I know that at the time the function is called: rcx has hex 1B00, which is the flag values I specified (dwFlags); rdx has hex 33, which is the bogus initial handle value that I plugged in to provoke the error (lpSource); r8 has 6, which is the error message number that was returned by GetLastError, which equates to ERROR_INVALID_HANDLE (dwMessageId); r9 has 0, for default language (dwLanguageId); rsp+32 has the address of msgptrp, which is my area to receive the address of the error message as allocated by the called function (lpBuffer); rsp+40 has hex 28, or decimal 40, which is the minimum number of characters I arbitrarily specified for the error message (nSize); and rsp+48 has the address of argmntsp, which according to the documentation should be ignored with the flags I specified (*Arguments).

If there's a problem, and obviously there is since rax is returned as zero, I suspect that it has to do with lpBuffer, which the documentation says has data type LPTSTR, whatever that is. Sometimes I want to shout, "Okay, that's what it is in terms of C++, but what is it really? I hope someone can easily spot where I ran off the rails, because I'm at my wits' end with this, and it's something I need to get working in order to do effective error checking for my future endeavors.

goterr    PROC
;
.data 
; flag values used:
; hex 00000100 FORMAT_MESSAGE_ALLOCATE_BUFFER
; hex 00000200 FORMAT_MESSAGE_IGNORE_INSERTS
; hex 00000800 FORMAT_MESSAGE_FROM_HMODULE
; hex 00001000 FORMAT_MESSAGE_FROM_SYSTEM
flagsp    dd 00001B00h ; those flags combined  

saveinitp dq     ?
bmaskp    dq     0fffffffffffffff0h
savshadp  dq     ?
msgptrp   dq     ?
handlep   dd     ?
argmntsp  dd     ?
;
.code
          mov    saveinitp, rsp    ; save initial contents of stack pointer in this proc         
          sub    rsp, 56           ; shadow space (max of 7 parameters * 8)
          and    rsp, bmaskp       ; make sure it's 16 byte aligned
          mov    savshadp, rsp     ; save address of aligned shadow area for this proc
;
          mov    handlep, ecx      ; save passed I/O handle value locally
;
          call   GetLastError      ; get the specific error
;
          mov    rsp, savshadp     ; shadow area for this proc
          mov    ecx, flagsp       ; flags for Windows error routine
          mov    edx, handlep      ; handle passed from caller
          mov    r8d, eax          ; msg id from GetLastError in low order dword
          mov    r9, 0             ; default language id
          lea    rax, msgptrp      ; pointer to receive address of msg buffer
          mov    [rsp+32], rax     ; put it on the stack
          mov    rax, 40           ; set lower doubleword to minimum size for msg buffer 
          mov    [rsp+40], rax     ; put it on the stack
          lea    rax, argmntsp     ; variable arguments parameter (not used)
          mov    [rsp+48], rax     ; put it on the stack
          call   FormatMessageA    ; if rax eq 0 the called failed, otherwise rax is no chars.
          mov    rsp, saveinitp    ; restore initial SP for this proc
          ret
goterr    ENDP
Robert Watson
  • 39
  • 4
  • Your `lpBuffer` seems fine. Verify your incoming `handlep` is valid. Not sure if handles are still 32 bits? I'd pass on the full 64 bits just to be sure. Also, call `GetLastError` **after** `FormatMessageA` returns with a zero to check the reason. – Jester May 04 '20 at 00:35
  • I should have specified that the flags I set are: FORMAT_MESSAGE_ALLOCATE_BUFFER, FORMAT_MESSAGE_IGNORE_INSERTS, FORMAT_MESSAGE_FROM_HMODULE, and FORMAT_MESSAGE_FROM_SYSTEM. – Robert Watson May 04 '20 at 00:36
  • 1
    You also need to declare [unwind codes](https://learn.microsoft.com/en-us/cpp/build/exception-handling-x64?view=vs-2019). This means using `PROC FRAME` instead of just `PROC` and adding directives like `.ALLOCSTACK` You are also required to put the frame in a frame pointer; you can't put it in a global variable. – Raymond Chen May 04 '20 at 00:38
  • No, the handle is not valid. I deliberately set it to an invalid value in order to provoke the error and test the error routine. But I will check GetLastError after the call. I probably should have thought of that. – Robert Watson May 04 '20 at 00:40
  • If the handle is not valid you should not set `FORMAT_MESSAGE_FROM_HMODULE` or pass a `NULL` instead of the invalid handle. – Jester May 04 '20 at 00:41
  • 2
    @Jester "*Not sure if handles are still 32 bits?*" - kernel handles are 64bit, but only the lower 32 bits are relevant. [32-bit and 64-bit Interoperability](https://learn.microsoft.com/en-us/windows/win32/winauto/32-bit-and-64-bit-interoperability) – Remy Lebeau May 04 '20 at 00:43
  • Maybe I'm missing the point here, but the object of the exercise was to get the formatted message stating that the handle is invalid. In a real life situation I might not already be aware that it's invalid, and that's why I would be trying to get the formatted message in the first place - to find out what the problem is. But in this case it's a make believe situation just to test the call to FormatMessageA. – Robert Watson May 04 '20 at 00:51
  • If CreateFile fails, you don't use the handle (that you didn't get) as a file handle in which to search for error message text. That's what you are doing here. If you want to ask the OS what OS error #x is, then just use FORMAT_MESSAGE_FROM_SYSTEM. – David Wohlferd May 04 '20 at 01:00
  • 2
    To emphasize: You aren't passing a handle to say "what's wrong with this handle?" The handle you are passing is a file that is formatted in such a way that Windows can retrieve formatted error messages from it, ones that it may not already have defined as system error messages. So your typical "readme.txt" file isn't in any way useful as a parameter here. – David Wohlferd May 04 '20 at 01:11
  • More generally: the functionality MS is trying to expose here is "application defined error messages." So while APPLICATION_ERROR_1 might mean "Invalid PartID" to one application, it could mean "Student already graduated" in another. Having a resource DLL that defines what APPLICATION_ERROR_1 means in your specific application is conceptually quite useful. But it doesn't sound like what you are trying to do here at all. If you don't have an actual resource dll with formatted error messages in it, don't use FORMAT_MESSAGE_FROM_HMODULE, like Jester said. – David Wohlferd May 04 '20 at 01:25
  • Also: LPSTR points to a null terminated ascii string. LPWSTR points to a null terminated unicode (aka wide-character) string. LPTSTR points to ascii if UNICODE is not #defined, else it points to unicode. Similarly, calling FormatMessage will call FormatMessageA if UNICODE is not defined and FormatMessageW if it is. Win9x was ascii by default, and calls to W functions got internally translated to the ascii version. NT (what we now call Windows) is the reverse. Allowing applications to "easily" support either was once very important. Now? Meh. – David Wohlferd May 04 '20 at 01:50
  • use global variable for save *rsp* very bad idea. and you not need `and rsp, bmaskp` - at function entry point `rsp == 16*n + 8`. and instead `call fn` use `call __imp_fn` for imported functions – RbMm May 04 '20 at 08:47
  • *"Okay, that's what it is in terms of C++, but what is it really?"* - Write a C program and have it output its preprocessed source code. Make sure to `#define MBCS` if you want to call `FormatMessageA`. Kudos to you if you do. Now your code can also run on Windows 95! If you find a 64-bit release of Windows 95, that is. – IInspectable May 04 '20 at 16:58
  • The online documentation for FORMAT_MESSAGE_FROM_STRING says: "Pointer to a string that consists of unformatted message text. It will be scanned for inserts and formatted accordingly." I took that to mean that I was supposed to provide the string of unformatted message text, which makes no sense if I don't know what the error is. Does it mean that the called function will provide the string? – Robert Watson May 05 '20 at 23:44
  • Some of the comments are more than I can comprehend, but I do understand one thing: If I'm trying to retrieve an error that resulted from a call to GetStdHandle, then it makes no sense to pass that handle to FormatMessageA, which is what I was doing. Maybe FORMAT_MESSAGE_FROM_STRING is a good replacement. – Robert Watson May 05 '20 at 23:47
  • It appears that when a valid handle is not available, so that flag FORMAT_MESSAGE_FROM_HMODULE can't be used when calling FormatMessageA, the alternative is FORMAT_MESSAGE_FROM_SYSTEM, not FORMAT_MESSAGE_FROM_STRING. – Robert Watson May 07 '20 at 01:25

0 Answers0