mmap does not return error nor valid pointer

Question

I've been trying to allocate memory with mmap, but so far I do not know how to do it properly. Below you can see a function I have written, it is called from C. The result it produces is always 197, which is the same as syscall number.

C declaration:

extern "C" void * ll_alloc ();

Assembly definition:

_ll_alloc:
  sub rsp, 8
  mov r9, 0
  mov rdi, 0
  mov rax, 197         
  mov rsi, 4096          
  mov r8, -1            
  mov rdx, 0x02 | 0x01
  mov rcx, 0x1000 | 0x0001
  syscall
  add rsp, 8
  ret

I called mmap directly from C and everything works as expected, call I made:

void * mem = mmap(NULL, 4096, PROT_WRITE | PROT_READ, MAP_ANON | MAP_SHARED, -1, 0);

I think MacOS has an equivalent of Linux `strace`, for tracing system calls. Use that to find out what happens with your `syscall`. I suspect that eax=197 is the wrong call number; I thought MacOS system calls had a high bit set. I find it surprising it would leave RAX unmodified, though, instead of returning `-ENOSYS` if that was the problem. — Peter Cordes, Jun 21 '19 at 23:27
Here is the line we are looking for, dumped from dtruss `mmap(0x0, 0x1000, 0x3, 0x1001, 0xFFFFFFFFFFFFFFFF, 0x0) = 0x100783000 0` — Mateusz Stompór, Jun 21 '19 at 23:33
That's from your working version? I meant `dtruss` your non-working version to see what system call you actually made with asm. — Peter Cordes, Jun 21 '19 at 23:40
What operating system are you programming for? If this is macOS, then you need to actually use the system call numbers and constants from macOS which are different from the Linux numbers. The calling convention might be different, too, but I'm not sure. — fuz, Jun 21 '19 at 23:45
@PeterCordes Sorry! No, it was not a call from assembly. I was mistaken. I cannot find the call I made. I am using syscalls from the site: https://sigsegv.pl/osx-bsd-syscalls/. And yes, I am aware of the fact that they differ from those used in linux. — Mateusz Stompór, Jun 21 '19 at 23:48

Peter Cordes · Accepted Answer · 2019-06-22T00:11:30.747

mov rcx, 0x1000 | 0x0001 can't be right: the syscall instruction itself destroys RCX (with the saved RIP).

On x86-64 Linux the calling convention for syscalls uses R10 for the 4th arg, in place of RCX in the function-calling convention arg order. (What are the calling conventions for UNIX & Linux system calls on i386 and x86-64)

I think OS X does the same thing (so mov r10d, 0x1000 | 0x1), but it's literally impossible for the kernel to read what was in user-space RCX before syscall, so it's definitely not that register. See also comments on 64-bit syscall documentation for MacOS assembly

Also, x86-64 MacOS uses different call numbers than x86-64 Linux. Although 197 is correct according to https://sigsegv.pl/osx-bsd-syscalls/ which does seem to be about x86-64. (And 197 isn't the Linux syscall number)

I thought I remembered the actual EAX value needing a higher bit set, like 0x2000xyz; maybe that's something you need to add. Update: yes, macOS 64-bit System Call Table explains that adding 0x2000000 to the call number is correct.

In general to debug this kind of thing

Single-step into the call to C library mmap call to see register values when the syscall instruction runs from the known-good library implementation.

Also use dtruss (MacOS) or strace on your hand-written asm program to see what the kernel thinks it's actually doing.

I've made the change, but it still returns 197. I use https://sigsegv.pl/osx-bsd-syscalls/ and https://opensource.apple.com/source/xnu/xnu-2782.20.48/bsd/kern/syscalls.master as references to check syscall numbers — Mateusz Stompór, Jun 21 '19 at 23:46
I am not able to edit my comment so I'm going to put the explanation here. It worked!!. I changed two things. I added 0x2000000 to 197 and used r10d to store 4th argument. Explanation of this magic `0x2000000`. It has been already pointed out in another thread that it is necessary. Link: https://stackoverflow.com/a/53905561/6091011 — Mateusz Stompór, Jun 22 '19 at 00:03
@MateuszStompór: heh, I was off by one zero in my memory of the hex constant you have to add. Glad you were able to find a reference for that. — Peter Cordes, Jun 22 '19 at 00:12

mmap does not return error nor valid pointer

1 Answers1

In general to debug this kind of thing