The major problem you're having with alink is that it doesn't have any 64-bit support, which is why you had to use nasm -fwin32
to generate 32-bit object code. The second problem is that you haven't specified an entry point. Frustrating, isn't it? I wasted a lot of time with different linkers myself.
If you want to do win64 assembly with nasm, I suggest using golink. It takes the light, fast, no-nonsense approach to linking. If you want to use functions from a DLL in your code, you don't need any library files -- GoLink can do all the linking using just the DLL files themselves. It'll even pull them off the system path, so you don't need to put anything in the same folder as your source code.
The next major problem you're having is that your example code isn't suited for Windows. Here's one that you can use to get started that won't crash when you run it:
; example64.s
; nasm -fwin64 example64.s
; golink /console example64.obj kernel32.dll msvcrt.dll
bits 64
default rel
extern GetStdHandle
extern WriteFile
extern ExitProcess
extern printf
section .data
message db 'Hello, World!',10,0
msglen equ $-message
written dq 1
section .text
global Start ; GoLink will use Start as the default entry point
Start:
; Use the C library to print our message
mov rcx, message
call printf
; Now try using the Windows API
mov rcx, -11
call GetStdHandle
; Use WriteFile to print our message again.
; Notice the calling convention for 64-bit Windows uses
; rcx, rdx, r8, and r9 for the first 4 non-floating point arguments
; and then the rest are pushed onto the stack.
mov rcx, rax ; HANDLE hFile
mov rdx, message ; LPCVOID lpBuffer
mov r8, msglen ; DWORD nNumberOfBytesToWrite
mov r9, written ; LPDWORD lpNumberOfBytesWritten
push qword 0 ; LPOVERLAPPED lpOverlapped
call WriteFile
mov rcx, 0
call ExitProcess
Assuming it's saved as example64.s, you can assemble and link it like so:
nasm -fwin64 example64.s
golink /console example64.obj kernel32.dll msvcrt.dll
Note that the reason we include kernel32.dll is for the Windows API calls (WriteFile, ExitProcess, GetStdHandle). Likewise, msvcrt.dll is for standard C library functions (i.e. printf, malloc, etc.). If you want to get really down and dirty with Win64 assembly, you'll probably want to go ahead and just use the Windows API, leaving out msvcrt.dll. You can find documentation for all the Windows API functions and data structures on MSDN.
Finally, it's worth noting that a lot of the function prototypes and structures they give on MSDN are for the 32-bit Windows API, so whenever you see a DWORD, you'll probably want to use a QWORD instead.
Anyway, I hope that gets you started in the direction you want to go. Good luck!