win32 very low level assembly - application startup issue

Question

I am busy programming a win32 program in assembly with a form and buttons... The problem is windows modify my variables in ram. The place were a store my hInstance and hwnd variables. I have found a workaround, but it is not an elegant solution. I would like to know why windows modify my variables and also were can I find documentation which describe the start up of an application.

MyWndProc: 
push EBP 
mov EBP, ESP 
mov eax, [EBP + 12] 
cmp eax, WM_DESTROY 
jne MyWndProc_j2 
push 0 
call PostQuitMessage 
jmp MyWndProc_j1 

MyWndProc_j2: 
cmp eax, WM_CREATE 
jne MyWndProc_j1 
mov eax, [EBP+8] 
push eax 
call CreateControls 
add esp, 4 

MyWndProc_j1: 
mov eax, [EBP + 20] 
push eax 
mov eax, [EBP + 16] 
push eax 
mov eax, [EBP + 12] 
push eax 
mov eax, [EBP + 8] 
push eax 
call DefWindowProcA 
pop EBP 
ret 

segment .data 

Wtitle db 'My Window',0 
ClassName db 'myWindowClass',0 

editClass db 'EDIT',0 
buttonName db 'OK',0 
buttonClass db 'BUTTON',0 
textName db 'My textbox',0 
textClass db 'edit',0 

formEdit db 'This is just a mem test', 0 

windowsVar1 dd 0 
windowsVar2 dd 0 
windowsVar3 dd 0 
windowsVar4 dd 0 
windowsVar5 dd 0 
windowsVar6 dd 0 
windowsVar7 dd 0 
windowsVar8 dd 0 

aMsg dd 0 
hwnd dd 0 
hwnd2 dd 0 
hwnd3 dd 0 
hInstance dd 0 
old_proc dd 0 
nCmdShow dd 0 
hfDefault dd 0

MyWndProc is the callback function from windows. At the 27'th call from windows, it modify the last 7 variables. If I switch the position of the last 8 variables with windowsVarx, then it still modifies hwnd, hwnd2 ... without modifying windowsVarx. Where x is from 1 to 8

CreateControls: 
push EBP 
mov EBP, ESP 

push 0 
push 0 
call GetModuleHandleA 
push eax 
push IDC_MAIN_BUTTON 
mov eax, [EBP+8] ;hwnd 
push eax 
push 24 
push 100 
push 220 
push 50 
mov eax, WS_CHILD 
or eax, BS_DEFPUSHBUTTON 
or eax, WS_TABSTOP 
or eax, WS_VISIBLE 
push eax 
push buttonName 
push buttonClass 
push 0 
call CreateWindowExA 
mov [hwnd2], eax 

push DEFAULT_GUI_FONT 
call GetStockObject 
mov [hfDefault], eax 

push 0 
mov eax, [hfDefault] 
push eax 
push WM_SETFONT 
mov eax, [hwnd2] 
push eax 
call SendMessageA 

push 0 
push 0 
call GetModuleHandleA 
push eax 
push IDC_MAIN_EDIT 
mov eax, [EBP+8] ;hwnd 
push eax 
push 100 
push 200 
push 100 
push 50 
mov eax, WS_CHILD 
or eax, ES_MULTILINE 
or eax, ES_AUTOVSCROLL 
or eax, ES_AUTOHSCROLL 
or eax, WS_VISIBLE 
push eax 
push 0 
push editClass 
push WS_EX_CLIENTEDGE 
call CreateWindowExA 
mov [hwnd3], eax 

push 0 
mov eax, [hfDefault] 
push eax 
push WM_SETFONT 
mov eax, [hwnd3] 
push eax 
call SendMessageA 

push Wtitle 
push 0 
push WM_SETTEXT 
mov eax, [hwnd3] 
push eax 
call SendMessageA 

pop EBP 
ret

The following function is the message loop, which collect and dispatch.

MyMessageLoop:
push 0
push 0
push 0
push aMsg
call GetMessageA
cmp eax, 0
je MyMessageLoop_j1
push aMsg
call TranslateMessage
push aMsg
call DispatchMessageA
jmp MyMessageLoop
MyMessageLoop_j1:
ret

Programming in assembly and assuming that it is the operating system that makes your code fail is a not productive. Use the debugger's data breakpoint feature. — Hans Passant, Feb 24 '12 at 13:00
@HansPassant, I think it's probably more ignorance than hubris. — Prof. Falken, Feb 24 '12 at 13:05
Do you mean between calls to the program, or after call to a function? But you really need to work on the question, for instance by putting in example code and results. — Prof. Falken, Feb 24 '12 at 13:06
Yeah, it's like this: which is more likely, an OS with millions of running installations and billions of running app instances has an until-now-undiscovered bug that corrupts user variables on app startup, or you have a bug in your app? — Martin James, Feb 24 '12 at 14:12
You all can stop your sarcastic comments. I never said it was a bug in windows, I ask why does windows modifies my variable is there documentation I can read? I have seen this in the debug memory window — andre, Feb 24 '12 at 15:02
Windows doesn't modify your variables. That's just absurd. What might happen is that you pass pointers to your variables to Windows and ask it to write to them. Which it would obediently do. Or your asm could be flawed. — David Heffernan, Feb 24 '12 at 15:14

score 1 · Answer 1 · answered Feb 24 '12 at 13:00

Your problem explanation isn't really clear. But you should remember that by calling a system call you may indeed end up with different values in your registers. I don't know about Windows, but on amd64 Linux, the kernel (which executes the system call) is required only to preserve the values of the registers r12 and up. The values in all the other registers may be changed and therefore will most probably not be the same after returning from the system call.

In order to remedy that, simply store the variables on your function's stack before calling the system.

It is a callback from windows, when the app starts up. It also happens when I store it in the my stack. It only modifies hwnd, hwnd2 ... No other variable gets modified — andre, Feb 24 '12 at 15:09

score 1 · Answer 2 · answered Feb 24 '12 at 15:25

1

It might appear that Windows is modifying your data, but as others have pointed out, it's more likely a bug or some other corruption in your code is causing problems.

It's almost impossible for people to determine the runtime behaviour of your entire program from snippets and programming in assembly almost always causes problems rarely seen when higher-level languages are used.

The best advice is to use a debugger and either step through the code or set a data breakpoint on the variables being modified. Data breakpoints are designed to stop your program on the instruction that performs the data modification.

You could also look at what the actual values of the data that are overwriting your variables - it might give you some clue as to where or why the memory is being overwritten.

The reason for people's sarcasm is that in your second sentence you are assuming Windows is to blame for your program not working. In many situations, blaming the Operating System is a good sign that the developer does not understand something or is reluctant to accept that they have made a mistake. The end result is almost always someone else pointing out the mistake.

answered Feb 24 '12 at 15:25

adelphus

10,116
5
36
46

I have used a debugger and I also stepped through the code, thank you for the debriefing on breakpoints. I know exactly which call corrupt my variables. It is in the user32.dll it is, int 2B. This does not happens when I link with gcc, only with alink. The purpose of this code is to learn assembly in the windows 32 environment. All I ask is why does it happens or some sort of documentation which explains this procedure, I am not here to blame windows. You make the assumption that I blame windows, it might appear this way. Like I said I have already a work around to fix my bug. – andre Feb 24 '12 at 19:46
@andre: When your data gets corrupted, what function are you calling in user32? – arx Feb 24 '12 at 20:13
After windows called my MyDwnProc it returns to user32. The ESI register contains the value of the address of the hwnd variable. I never used the ESI register. A lot of stuff happens in the user32 until it reach the call, int 2B. This happens after the 27'th return – andre Feb 24 '12 at 20:50
@andre: What message was passed to your wndproc before the failure? – arx Feb 24 '12 at 21:06
It is 7F which is equal to WM_GETICON? – andre Feb 24 '12 at 21:26
You are meant to preserve ESI. If you don't preserve it then your code is faulty. You are allowed to modify EAX, EDX, and ECX. Preserve all other registers. – David Heffernan Feb 24 '12 at 22:27
You DO NOT have to preserve ESI, EDI, EBX if and only if you do not use them in a windows callback proc. If in your CreateControls proc, you modify esi, edi, or ebx then you must preserve them since they interact with a callback proc. – Gunner Feb 25 '12 at 03:48
I only use EBX, which is preserved. I also load other programs into ollydbg which were also link with Alink. It seems that there are also changes in the memory. It does not effect the program, because that part of the memory is only being used once. I do not think the problem is with alink, I still think it is a bug in the assembly code, I just don't why this strange behavior occurs. – andre Feb 25 '12 at 11:28
@Gunner What you have described is what is known as preserving. You just need to make sure that they hold the same value on return as they did on entry. – David Heffernan Feb 25 '12 at 13:05
@andre Is ESI being modified during one of your functions? – David Heffernan Feb 25 '12 at 13:07
No, I do not use ESI. ESI has the same return value as its entry value. – andre Feb 25 '12 at 20:45

score 0 · Answer 3 · answered Feb 25 '12 at 18:41

0

Why are you adding 4 to esp after a call to CreateControls? You are pushing 1 dword onto the stack, does CreateControls not cleanup the stack itself? You wrote that function? If it does adjust the stack with something like ret 4*1 then that add esp, 4 is screwing everything up. That snippet of code does nothing for us, plus there are so many uneeded movs and jmps there

@David Heffernan thanks I just started writing all my programs in Assembly over 10 years ago and well aware of how and what registers need to be preserved and when you DO NOT need to save them, just because a complier saves all registers in the prologue, doesn't mean it's correct.

answered Feb 25 '12 at 18:41

Gunner

5,780
2
25
40

I have checked the stack is fine. CreateControls is my function. Is it my responsibility to preserve the registers that I do not use? – andre Feb 25 '12 at 20:51
How are you checking that the stack is fine? Post the code for the CreateControls function so we can check it. – Gunner Feb 25 '12 at 21:54
I checked with OllyDbg. I can see the stack in the stack window. – andre Feb 26 '12 at 15:31

score 0 · Accepted Answer · answered Feb 26 '12 at 18:20

Where do I start? Your code formatting is very hard to read. Hope a Uni is not teaching you it. Anyways, your WindowProc was VERY wrong. After EVERY message you handle, you DO NOT call DefWindowProc, most of the messages you just return 0 in eax.

After your call to CreateControls, you don't need add esp, 4 as long as at the end of CreateControls you do ret 4 or ret 4 * NumOfParamsPassed.

I fixed your WindowProc and the window now shows. I also removed the many unneeded movs.

MyWndProc:
    push    ebp
    mov     ebp, esp

    mov     eax, dword ptr[ebp + 3 * 4] ; same as [ebp + 12]
    cmp     eax, WM_CREATE
    je      _CREATE
    cmp     eax, WM_CLOSE
    je      _CLOSE

PassThrough:    
    push    dword ptr[ebp + 5 * 4]; same as [ebp + 20]
    push    dword ptr[ebp + 4 * 4]; same as [ebp + 16]
    push    dword ptr[ebp + 3 * 4]; same as [ebp + 12]
    push    dword ptr[ebp + 2 * 4]; same as [ebp + 8]
    call    DefWindowProc
    jmp     _DONE

_CLOSE:
    push    0
    call    PostQuitMessage
    jmp     _RET0

_CREATE:
    push    dword ptr[ebp + 2 * 4]
    call    CreateControls
    ;add     esp, 4

_RET0:
    xor     eax, eax

_DONE:
    pop     ebp
    ret     4 * 4 ; <----- you were missing this!!!!!

Thanx for this code, I really appreciate the effort. The real bug in the system was aMsg, it is one of my variables. I did not realise that the size of the message in the message loop will exceeds 4 bytes. It was actually an overflow that occurred which appears that windows were modifying my variables which was next to aMsg. Thanx for all the help, I will code more efficient from now on, it was because of my C background. — andre, Feb 26 '12 at 22:07
You didn't know that aMsg in your message loop should be a pointer to a MSG structure? It is the same in C, no? — Gunner, Feb 26 '12 at 22:17
The reason that I did not pickup the problem in the first place is that the first byte of aMsg did not change during this procedure. — andre, Feb 26 '12 at 22:20
I did not know it was a pointer to a structure. The C I mean my coding standard in assembly. If else etc... — andre, Feb 26 '12 at 22:23

win32 very low level assembly - application startup issue

4 Answers4