2

I'm trying to use an Assembly(FASM) program as a function in C++, but it doesn't seem to be working, even though I linked the compiled object. Any advice?

C++ program:

#define _CRT_SECURE_NO_WARNINGS
#include <iostream>
#include <stdio.h>
#include <string.h>

using namespace std;
extern "C"
{
    unsigned int __stdcall long_add(int*, int*, int*);
}

int main()
{
    int testOne[3] = { 0x12345678, 0x04000000, 0xFFFFFFFF };
    int testTwo[3] = { 0x12345678, 0x04000000, 0xFFFFFFFF };
    int testOut[3] = {};

    long_add(testOne, testTwo, testOut);

    scanf;
}

Assembly function:

include 'include\win32a.inc'

format MS COFF 

public long_add as '_long_add@32'

section '.code' code readable executable

proc long_add IN_NUM1, IN_NUM2, OUT_NUM

start:

        mov ecx, 3
        clc
        
ADDING:

        mov eax, [IN_NUM1+(ecx-1)*4]
        mov ebx, [IN_NUM2+(ecx-1)*4]
        adc eax, ebx
        mov [OUT_NUM+(ecx-1)*4], eax
        loop ADDING
        
        ret
        endp

Link Proof: https://i.stack.imgur.com/au0jy.png

Edit: I tried to moving array addresses into registers, and ended up with the code below:

format MS COFF 

public long_add as '_long_add@12'

section '.code' code readable executable

proc long_add IN_NUM1, IN_NUM2, OUT_NUM

start:

        mov ecx, 3
        clc
        mov edi, [IN_NUM1]
        mov esi, [IN_NUM2]
        
ADDING:

        mov eax, [edi+(ecx-1)*4]
        mov ebx, [esi+(ecx-1)*4]
        adc eax, ebx
        push eax
        loop ADDING
        
        mov esi, [OUT_NUM]
        mov ecx, 3
        
POPPING:
        
        pop eax
        mov [esi], eax
        inc esi
        loop POPPING
        
        ret
        endp

And while this code "works", as in the program doesn't crash, when I try to add {0x12345678, 0x04000000, 0xFFFFFFFF} and { 0x87654321, 0x02000000, 0x00000001 }, it returns {409, 0, 0}.

Maxim Khannaov
  • 41
  • 4
  • 3
    It is looking for `...@12` and you export the symbol as `...@32`. – Botje Nov 03 '20 at 09:25
  • 2
    According to [The history of calling conventions](https://devblogs.microsoft.com/oldnewthing/20040108-00/?p=41163) that number denotes the bytes passed as arguments, so 32 is not possible for three pointers. – Botje Nov 03 '20 at 09:31
  • 2
    Also, as well as that symbol-name bug, you'll need to fix `ret 12`. Verifying agreement between caller and callee for this caller-pops-the-stack calling convention is the whole point of decorating function names with `@12` or `@4`. – Peter Cordes Nov 03 '20 at 09:39
  • @PeterCordes I have changed '_long_add@32' to '_long_add@12' in the Assembly code, and while my program doesn't crash on start anymore, it doesn't change the contents of arrays. Also, what do you mean by `ret 12`? I don't have that in my code. – Maxim Khannaov Nov 03 '20 at 17:10
  • 1
    Exactly, you just have `ret` which fails to pop the caller's args, violating the `stdcall` calling convention. https://en.wikipedia.org/wiki/X86_calling_conventions#Callee_clean-up – Peter Cordes Nov 03 '20 at 23:28
  • 1
    Given your `proc long_add IN_NUM1, IN_NUM2, OUT_NUM` declaration, you're probably just modifying the stack (which holds pointers), not the data pointed-to. You need to load those pointers into registers so you can deref them; x86 doesn't have memory-indirect addressing modes. Use a disassembler to see what machine instructions you actually got out of MASM: it's "high level" stuff can be the opposite of helpful when you don't yet understand what exactly will happen when you use it this way. – Peter Cordes Nov 03 '20 at 23:31
  • @PeterCordes After following your advice, my code works works fine as a program, but doesn't compile as a function (see edit). What could be causing this? – Maxim Khannaov Nov 04 '20 at 10:10
  • 1
    `proc long_add IN_NUM1, IN_NUM2, OUT_NUM` doesn't assemble at all in fasm for GNU/Linux, it just says `illegal instruction` for that line. So I can't test it, and I don't know FASM particularly well. I thought only MASM / TASM used that syntax with magic names for operands instead of referencing the stack explicitly. – Peter Cordes Nov 04 '20 at 10:21
  • @PeterCordes Sorry, I missed some lines at the top of the code, I hope this is better. Also, I just needed to put the input values into square brackets to get the code to compile. Still can't get the proper output though. – Maxim Khannaov Nov 04 '20 at 10:53
  • 1
    Unfortunately `fasm` 1.73.24 for GNU/Linux still doesn't assemble that file; same result. Perhaps when built for Windows, it supports different source syntax? That would be weird, but I don't know FASM very well. – Peter Cordes Nov 04 '20 at 11:08

1 Answers1

1

The comments already pointed out the stdcall naming mistake. What's causing the wrong output is that you use inc esi to loop through OUT_NUM, while you should use add esi, 4, like you did in the addition loop, because each int is 4 bytes. Alternatively, you could use stosd, or store directly in the addition loop.

I would also recommend pushing esi&edi at the start of the function and popping them at the end (reverse order!), because I'm pretty sure that these are non-volatile (callee-saved) registers, so the C compiler may assume they are not modified.

What the comments are telling you about replacing ret with ret 3*4 is not true, because the proc macro in stdcall mode does this for you automatically.

SWdV
  • 1,715
  • 1
  • 15
  • 36