CreateFileA, CreateFileW don't work in masm64 (Russian Variant)

Question

I'm using a Russian variant of the MASM64 SDK that can be found here. My code is:

OPTION DOTNAME

option casemap:none

include G:\Programs\Soft\Coding\MASM\masm64\Include\win64.inc
include G:\Programs\Soft\Coding\MASM\masm64\Include\kernel32.inc
include G:\Programs\Soft\Coding\MASM\masm64\Include\user32.inc
include G:\Programs\Soft\Coding\MASM\masm64\Include\ADVAPI32.inc
include G:\Programs\Soft\Coding\MASM\masm64\Include\temphls.inc
include G:\Programs\Soft\Coding\MASM\masm64\Include\unicode_string.inc

includelib G:\Programs\Soft\Coding\MASM\masm64\lib\user32.lib
includelib G:\Programs\Soft\Coding\MASM\masm64\lib\kernel32.lib
includelib G:\Programs\Soft\Coding\MASM\masm64\lib\comctl32.lib
includelib G:\Programs\Soft\Coding\MASM\masm64\lib\gdi32.lib
includelib G:\Programs\Soft\Coding\MASM\masm64\lib\advapi32.lib

.data
nFILE           db 'OSPFv3.txt', 0
.code
WinMain proc 
    ENTER 28, 0
    MOV R11, RDI
    AND SIL, 0C1h
    ROR R8W, 0F8h
    ROL R8B, 09Ah
    ADD RDI, 08h
    NOT RSI
    MOV SIL, AL
    JMP @R11_1
    MOV R8, R11
@R11_1:    
      invoke CreateFile,&nFILE, GENERIC_READ or GENERIC_WRITE, NULL, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL;Создаю файл
      invoke ExitProcess, NULL
WinMain endp
end

CreateFileA doesn't work in this code, and CreateFileW doesn't work either.

This is a screenshot using an ANSI string:

This is a screenshot using a Unicode string:

It also gives me that my parameters are incorrect. Why, when they all appear to be correct?

EDIT with revised example code:

.data
Ascii_1         db "OSPFv3.txt", 0
Unicode:        du <OSPFv3.txt>, 0
.code
WinMain proc 
      push rbp
      mov rbp,RSP
      sub rsp, 28h 
      invoke CreateFile,&Ascii_1, GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ or FILE_SHARE_WRITE, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL;Создаю файл
      invoke ExitProcess, NULL
WinMain endp
end

I have no idea whether it's the specific cause, but I'll just note that RSP is misaligned — harold, Sep 21 '22 at 00:07
What's the point of those and/rotate/add/not instructions at the top of WinMain? You're using `invoke`, so other register values don't matter, only stack alignment. As harold notes, RSP isn't aligned because you used `28`, not `28h`, with `enter` (which is very slow and usually shouldn't be used vs. `push rbp`/`mov rbp,rsp`/`sub rsp, 28h`). — Peter Cordes, Sep 21 '22 at 00:12
(I wasn't sure `invoke` even worked in 64-bit MASM, or maybe that was only FASM where they don't provide it because of shadow space being required, at least in functions that are going to return. Here, you pass more than 4 args so stores to write stack args will overwrite space above the shadow space, even if you'd used `28h`, as we can see in your disassembly..) — Peter Cordes, Sep 21 '22 at 00:13
@PeterCordes : 64-bit MASM doesn't provide `invoke` anymore but the MASM64 SDK provides one as a macro http://masm32.com/board/index.php?topic=10052.0 — Michael Petch, Sep 21 '22 at 00:18
I'm sorry I didn't answer earlier. I edited the above post Is that better? The problem remains the same... — OSPFv3, Sep 21 '22 at 02:20
Russian guys made macros for masm64 here is the folder. http://dsmhelp.narod.ru/masm64.zip I will be glad if you help me further! I also made a macro for unicode_string, above I have it included in .asm ;---- I am writing a VM in assembler, this is the handler that I log, you can see it in the instructions above. — OSPFv3, Sep 21 '22 at 02:25
[Please do not upload images of code/data/errors when asking a question.](//meta.stackoverflow.com/q/285551). Images should only be used when there is no other way to demonstrate an issue. Also, on the occasion that an image is needed (which is not the case here), you are required to place the image in SO's imgur account so that it is permanent. Off-site links to images are not acceptable. — Ken White, Sep 21 '22 at 02:38
I know what is going on (mostly). There appear to be some different versions of MASM64 out there. Is there a reason you don't use the MASM64 version from the MASM32 forum? A link to the zip file is in this post: http://masm32.com/board/index.php?topic=10052.0 . You can download it from this link: http://www.masm32.com/download/masm64.zip . Are you required to use version from the Russian site? If you can try the MASM32's version you may find it handles the stack in a more straight forward way and its version of `invoke` may be easier to use/understand. — Michael Petch, Sep 21 '22 at 04:46
If you want to use the MASM32 version of MASM64 then you can replace all your include lines by including `masm64rt.inc` — Michael Petch, Sep 21 '22 at 04:49
As well if you use the MASM32 forum version of MASM 64 you have to remove the `push rbp` `mov rbp,RSP` `sub rsp, 28h` . It should be properly generated for you. — Michael Petch, Sep 21 '22 at 05:05
If you want to use the Russina version, per the Russian docs you need to tell the PROC the maximum number of arguments used by Invoke within the function. You do that by using `<#>` where `#` is the maximum number of arguments to Invoke. I also discovered you needed special PROLOGUE/EPILOGUE definition and I found a bug that you need at least one LOCAL variable for things to work properly (for RBP to be properly initialized). A version of the code that seems to work with the Russian version is here: https://pastebin.com/mSc4RRcc — Michael Petch, Sep 21 '22 at 07:33
Thank you, I understand you (Ken White) about the pictures. I apologize I am responding to your first comment after Ken White, Michael Petch. I am not using the version from this site(masm32.com/download/masm64.zip ), since it does not download from me ... About your second and third comments, I understand you. Special thanks for the code!!!!!!!!!!!!!!!!! You are a masm genius! — OSPFv3, Sep 21 '22 at 07:49
Yeah, I found the masm64 version on the masm32 forum doesn't download in Chrome. I had to download it manually outside a browser and then scanned it for viruses and there were none. It is almost like Chrome has that site blacklisted. — Michael Petch, Sep 21 '22 at 07:52
The MASM64 on the MASM32 forum seems to just work and doesn't need a dummy variable, the different epilogue/prologue options aren't needed, and doesn't require (or even support) the `<#>` on `PROC`. The generated code just seems to work better without any hoops, although the stack usage may be worse (less efficient) than the Russian version. — Michael Petch, Sep 21 '22 at 08:02
I discovered that the bug occurs if there are neither parameters listed for a PROC (procedure), nor are there any local variables declared. I found that if you just define the argument list with `varargs` there is no need for the dummy variable. So this also seems to work `WinMain proc <7> varargs` (the dummy variable can be removed) — Michael Petch, Sep 21 '22 at 08:58
Thank you for your help!!!!!!!!!!! Everything was downloaded via FireFox! Special thanks for the addition of information about WinMain! — OSPFv3, Sep 21 '22 at 09:31

CreateFileA, CreateFileW don't work in masm64 (Russian Variant)

0 Answers0

Linked