Simple encryption Assembly Program - Access violation writing to memory location

Question

I had to implement a cdecl calling convention into this program which originally used a non standardized convention. As far as I can tell it looks right, but I get a unhandled exception error saying "Accress violation writing location 0x00000066, which seems to hit when the program gets down to the line "not byte ptr[eax]" or atleast that is where the arrow points after breaking the program.

Could anyone tell me what is wrong with my program and how I may fix it? Thank you.

void encrypt_chars (int length, char EKey)
{   char temp_char;                     

for (int i = 0; i < length; i++)    
{
    temp_char = OChars [i];         
    __asm {
            push   eax
            movzx  eax, temp_char
            push   eax
            lea    eax, EKey
            push   eax
            call   encrypt
            mov    temp_char, al

            pop    eax
    }
    EChars[i] = temp_char;          
return;


// Inputs: register EAX = 32-bit address of Ekey,
//                  ECX = the character to be encrypted (in the low 8-bit field, CL).
// Output: register EAX = the encrypted value of the source character (in the low 8-bit field, AL).

__asm {          

encrypt:

        push ebp
        mov ebp, esp
        mov ecx, 8[ebp] 
        mov eax, 12[ebp]
        push edi                  
        push ecx                  
        not byte ptr[eax]         
        add byte ptr[eax], 0x04   
        movzx edi, byte ptr[eax]  
        pop eax                   
        xor eax, edi              
        pop edi                   
        rol al, 1                 
        rol al, 1                 
        add al, 0x04              
        mov esp, ebp
        pop ebp
        ret                       
}

Answer 1

By inspection, the comment on the encrypt function is wrong. Remember: the stack grows down, so when the arguments are pushed onto the stack, the ones pushed first have the higher address and, therefore, the higher offset from the base pointer in the stack frame.

The comment to encrypt says:

// Inputs: register EAX = 32-bit address of Ekey,
//                  ECX = the character to be encrypted

However, your calling sequence is:

movzx  eax, temp_char    ; push the char to encrypt FIRST
push   eax
lea    eax, EKey         ; push the encryption key SECOND
push   eax
call   encrypt

So the character is push first. So the character to encrypt But encrypt is loading them this way:

; On function entry, the old Instruction Pointer (4 bytes) is pushed onto the stack
; so now the EKey is +4 bytes from the stack pointer
; and the character is +8 bytes from the stack pointer
;

push ebp
mov ebp, esp

; We just pushed another 4 bytes onto the stack (the esp register)
; and THEN we put the stack pointer (esp) into ebp as base pointer
; to the stack frame.
;
; That means EKey is now +8 bytes off of the base pointer
; and the char to encrypt is +12 off of the base pointer
;
mov ecx, 8[ebp]            ; This loads EKey pointer to ECX
mov eax, 12[ebp]           ; This loads char-to-encrypt to EAX

The code then proceeds to try to reference EAX as a pointer (since it thinks that's EKey), which is going to cause an access violation since it's your character to encrypt the first time it tries to reference EAX as a pointer, which is here:

not byte ptr[eax]

So your debugger pointer was right! :)

You can fix it just by swapping these two registers:

mov eax, 8[ebp]            ; This loads EKey pointer to EAX
mov ecx, 12[ebp]           ; This loads char-to-encrypt to ECX

Finally, your call to encrypt doesn't clean up the stack pointer when it's done. Since you pushed 8 bytes of data onto the stack before calling encrypt, and since encrypt does a standard ret with no stack clean-up, you need to clean up after the call:

...
call   encrypt
add    esp, 8
...