CreateProcessA function doesn't work in MASM64 (ml64.exe)

Question

I want to write program which starts another program (process). I am using MASM64 (ml64.exe) from Visual Studio 2015.

Program doesn't work. Nothing is shown. In debugger I get Access Violation.

I have no idea what is wrong with my code.

My code:

extrn ExitProcess : proc
extrn MessageBoxA : proc
extrn CreateProcessA : proc

PROCESS_INFORMATION    struct 
hProcess    DWORD    ?
hThread    DWORD    ?
dwProcessId    DWORD    ?
dwThreadId    DWORD    ?
PROCESS_INFORMATION    ends

STARTUPINFOA    struct 
cb    DWORD    ?
lpReserved    DWORD    ?
lpDesktop    DWORD    ?
lpTitle    DWORD    ?
dwX    DWORD    ?
dwY    DWORD    ?
dwXSize    DWORD    ?
dwYSize    DWORD    ?
dwXCountChars    DWORD    ?
dwYCountChars    DWORD    ?
dwFillAttribute    DWORD    ?
dwFlags    DWORD    ?
wShowWindow    WORD    ?
cbReserved2    WORD    ?
lpReserved2    DWORD    ?
hStdInput    DWORD    ?
hStdOutput    DWORD    ?
hStdError    DWORD    ?
STARTUPINFOA    ends

.const
MB_ICONINFORMATION equ 40h
ERROR_ALREADY_EXISTS equ 0B7h
NORMAL_PRIORITY_CLASS equ 020h

.data
szText db "This is first application which creates new process using CreateProcessA.", 00h
szCaption db "Information",00h
processInfo PROCESS_INFORMATION <>
startupInfo STARTUPINFOA <>
szProcName db "D:\Apps\SampleApp.exe", 00h

.code
    Main proc
        ;not sure if correct - begin
        lea rax, processInfo
        lea rbx, startupInfo


        sub rsp, 60h 
        push rax
        push rbx
        push 00h
        push 00h
        push NORMAL_PRIORITY_CLASS
        push 00h
        mov r9, 00h
        mov r8, 00h
        mov rdx, 00h
        lea rcx, szProcName
        call CreateProcessA
        add rsp, 60h 
        ;not sure if correct - end

        sub rsp, 28h
        mov r9, MB_ICONINFORMATION
        lea r8, szCaption
        lea rdx, szText
        xor rcx, rcx
        call MessageBoxA
        add rsp, 28h

        Exit:
        xor rcx, rcx
        call ExitProcess
    Main endp
end

build.bat

@echo off

ml64.exe prog1.asm /link /entry:Main /subsystem:windows /defaultlib:"kernel32.Lib" /defaultlib:"user32.Lib"

pause

Thanks in advance for Your help.

stack pointer must be aligned on 16 bytes, before any API call, but you not do this before call CreateProcessA - already enough for crash — RbMm, Jul 17 '16 at 11:13
I have there `sub rsp, 60h`. I have changed this to `sub rsp, 58h` but it still doesn't work... What value should be there? — Dave, Jul 17 '16 at 11:26
`In debugger I get Access Violation.` On which instruction? The whole point of using a debugger is to get detailed info and investigate further (e.g. examine register contents). — Peter Cordes, Jul 17 '16 at 12:13
in version with sub rsp,60h obviously crash by wrong stack alignment (16*n - 8 - 6*8 -> this is not 16 byte aligned) after fix that - not clear are still crash or simply not working, but it and must not work how minimum by not initialized size of startup info — RbMm, Jul 17 '16 at 12:21
Although it will not solve your immediate problem, you *really* should be calling `CreateProcessW`, rather than `CreateProcessA`. The ANSI functions (suffixed with an `A`) have been deprecated for many years. Always prefer the Unicode versions, suffixed with a `W` (for "Wide"). On Windows, Unicode == UTF-16. — Cody Gray - on strike, Jul 18 '16 at 07:40

score 4 · Answer 1 · answered Jul 17 '16 at 13:33

4

You should check definitions for PROCESS_INFORMATION and STARTUPINFO structures because they may differ between x86 and x64. For example, handles are defined as pointers, not DWORDs (32-bit integers).

answered Jul 17 '16 at 13:33

Martin Drab

667
4
6

score 4 · Answer 2 · edited Jul 18 '16 at 07:38

Besides initializing STARTUPINFO and being sure to use the correct data types (pointers are 64-bits wide in 64-bit Windows, while DWORD is always 32-bits), you also need to correctly allocate the Parameter Area (also sometimes known as the "Shadow Space"):

The parameter area is always at the bottom of the stack (even if alloca is used), so that it will always be adjacent to the return address during any function call.
It contains at least four entries, but always enough space to hold all the parameters needed by any function that may be called.
Note that space is always allocated for the register parameters, even if the parameters themselves are never homed to the stack; a callee is guaranteed that space has been allocated for all its parameters.

^{Emphasis mine, from here}

Assuming a correct initialization of all the data structures required, a possible invocation of CreateProcessA is:

; Stack is assumed aligned here

push rax                     ; Not aligned (08h)
push rbx                     ; Aligned     (10h)
push 00h                     ; Not aligned (18h)                    
push 00h                     ; Aligned     (20h)
push NORMAL_PRIORITY_CLASS   ; Not aligned (28h)
push 00h                     ; Aligned     (30h)

; Make room for the first four (register) parameters
; Stack is aligned, not need to subtract 28h, just 20h (4*8)

sub rsp, 20h

mov r9, 00h                  
mov r8, 00h
mov rdx, 00h
lea rcx, szProcName
call CreateProcessA

add rsp, 50h

Note that the caller is responsible for cleaning up the stack after the call.

You idea of reserving space on the stack with a sub rsp, ... is not entirely wrong.
Of course you have to do the math correctly, but most importantly that technique is not immediately compatible with the pushes.

Once you have sub rsp, ... one or more indirect stores of the kind mov [rsp+...], ... are required to set up the arguments. The pushes will just move the stack pointer again, making all the previous work useless.

The "parameter area" is often called "shadow space". Well spotted that the OP's code isn't following the win64 calling convention. — Peter Cordes, Jul 17 '16 at 22:13

score 0 · Answer 3 · answered Aug 31 '16 at 20:19

In your function reserve enough space for calling sub functions with so many Parameters. Use .ALLOCSTACK for that in your function prolog

Then simply assign the entire parameter list

mov QWORD PTR [rsp+48h], rax                     
mov QWORD PTR [rsp+40h], rbx                     
mov QWORD PTR [rsp+38h], 00h                     
mov QWORD PTR [rsp+30h], 00h                     
mov QWORD PTR [rsp+28h], NORMAL_PRIORITY_CLASS
mov QWORD PTR [rsp+20h], 00h

xor r9, r9    ; pass 0              
xor r8, r8    ; pass 0
xor edx, edx  ; pass 0  (higher DWORD becomes always also zero, saving the REX-byte)
lea rcx, szProcName
call CreateProcessA

score 0 · Answer 4 · answered Jan 25 '21 at 20:49

It's a bit old question but since I've just solved it writing it down. Now I was struggling with the same problem but with NASM. The issue is the same but the syntax will be a little bit different. Your problem is that your STARTUPINFOA and PROCESS_INFORMATION are incorrect for 2 reasons:

Pointers are DWORD64 on x64 system
You've not considered structure padding on 64bit systems

The correct structured in NASM syntax are here:

; https://msdn.microsoft.com/library/windows/desktop/ms686331.aspx
STRUC _STARTUPINFOA
    .cb:                resq 1
    .lpReserved:        resq 1
    .lpDesktop:         resq 1
    .lpTitle:           resq 1
    .dwX:               resd 1
    .dwY:               resd 1
    .dwXSize:           resd 1
    .dwYSize:           resd 1
    .dwXCountChars:     resd 1
    .dwYCountChars:     resd 1
    .dwFillAttribute:   resd 1
    .dwFlags:           resd 1
    .wShowWindow:       resd 1
    .cbReserved2:       resd 1
    .lpReserved2:       resq 1
    .hStdInput:         resq 1
    .hStdOutput:        resq 1
    .hStdError:         resq 1
ENDSTRUC

; https://msdn.microsoft.com/library/windows/desktop/ms684873.aspx
STRUC _PROCESS_INFORMATION
    .hProcess:          resq 1
    .hThread:           resq 1
    .dwProcessId:       resd 1
    .dwThreadId:        resd 1
ENDSTRUC

A little bit explanation:

resq = DWORD64
resd = DWORD32

If one follows the struct most of the fields can be filled properly but the are couple of fields which differ. The reason is the struct padding. MS compiler chooses the biggest element in the struct and then pads all other fields to it. In order to give an example STARTUPINFOA case looks like the following:

The compiler chooses DWORD64 as the biggest element
cb field is DWORD but since it can't pad it with the next field (lpReserved: DWORD64) it pads it to DWORD64
lpReserved, lpDesktop and lpTitle are already DWORD64
From dwX until dwFlags the size can be padded with the next element so no change
wShowWindow and cbReserved2 are only WORD so compiler pads them together to 8 bytes so each of the fields are changed to DWORD
From lpReserved2 until hStdError are already DWORD64

When I've done the correct padding it worked like charm. Good luck! :)

I've never heard of the term "DWORD64". A dword (doubleword) is always 32 bits wide on 8086-based architectures. A 64-bit number is a qword (quad word). — ecm, Jan 25 '21 at 20:53
Then please open visual studio and check "basestd.h": `typedef unsigned __int64 DWORD64, *PDWORD64;` — Gabor Somogyi, Jan 26 '21 at 07:51

CreateProcessA function doesn't work in MASM64 (ml64.exe)

4 Answers4

Linked