4

I'm loading a plain statically linked ELF binary in MIPS (32-bit). After mapping the loadable segments, this is how I set up the stack before jumping into my target ELF's CRT's _start function:

__asm__("                                                             \
      addi       $2,         %[envN],      0                         ;\
    .env_loop:                                                        \
      addi       $2,         $2,          -4                         ;\
      lw         $3,         0($2)                                   ;\
      addi       $sp,        $sp,         -4                         ;\
      sw         $3,         0($sp)                                  ;\
      bne        $2,         %[env0],      .env_loop                 ;\
      addi       $2,         %[argN],      0                         ;\
    .arg_loop:                                                        \
      addi       $2,         $2,          -4                         ;\
      lw         $3,         0($2)                                   ;\
      addi       $sp,        $sp,         -4                         ;\
      sw         $3,         0($sp)                                  ;\
      bne        $2,         %[arg0],      .arg_loop                 ;\
      addi       $2,         %[argc],      0                         ;\
      addi       $sp,        $sp,         -4                         ;\
      sw         $2,         0($sp)                                  ;\
      addi       $2,         %[func],      0                         ;\
      jr         $2                                                  ;"
    :
    : [envN] "r" (envp + envc + 1),
      [env0] "r" (envp),
      [argN] "r" (argv + argc + 1),
      [arg0] "r" (argv),
      [argc] "r" ((int32_t)argc),
      [func] "r" (entry_point)
    : "$2", "$3", "cc", "memory"
);

So that I'm pushing the environment variables, the command line arguments, argc on the stack, and finally jumping out to the target ELF's entry point. This works properly and I end up inside my loaded program's main function with the right command-line arguments and everything, except for one thing: malloc does not work! Any call to it returns null and sets errno to ENOMEM.

The MIPS emulator I'm using (qemu-system-mips) has plenty of memory free, and if I just launch the program without using my loader it works fine, so it must be coming from the loader. But I have no idea why; I used the same loading technique with x86, x86_64, and arm, and they all work great, but for some reason malloc is malfunctioning with the MIPS version of the loader.

Is there something I've missed here? Something that actually needs to be done prior to jumping into the loaded executable, that might matter for MIPS but not for the other architectures I've successfully tried? I figured I'd ask here to see if someone has come across this before, because I really can't imagine what's going wrong here.

I'm running this under Linux and using musl as libc. After some debugging I found out the difference occurs in musl's expand_heap function, but I don't have source to assembly information yet so it's not clear what the bug is (musl uses the same malloc code for the other architectures, and they work just fine).

The binary is linked statically (including the libc; it has no dynamic dependencies) and everything else (printf, fopen, etc..) works fine in the loaded binary with the apparently sole exception of malloc (and of course realloc/calloc, and I suppose free). So it's really puzzling.

Thomas
  • 3,321
  • 1
  • 21
  • 44
  • Did you statically link the `libc`? – user35443 Jun 07 '15 at 09:00
  • @user35443 Yes, the binary is statically linked and has the libc statically linked as well (the loader also has its own copy of the libc, since it needs do to some mprotect/mmap work, but the two copies are in separate address spaces) – Thomas Jun 07 '15 at 09:02

2 Answers2

3

It's been a while since I tried to do MIPS stuff, but I think your aux vector needs to look like this:

// main()'s pseudo arguments.
#define AT_PAGESZ 6
argv:
        .word   name
        .word   0                       // End of argv.
        .word   0                       // End of envp.
        // Auxv
        .word   AT_PAGESZ
        .word   4096                    // Page size.
        .word   0

As I recall, the page size auxv entry was unique to the MIPS for musl.

Richard Pennington
  • 19,673
  • 4
  • 43
  • 72
  • It works perfectly! This is exactly the kind of answer and expertise I was hoping for when I asked this question, thanks so much. You saved my day. :) – Thomas Jun 08 '15 at 00:18
  • @Thomas I've been working on using clang and musl in a baremetal environment. I'm working on ARM right now, but I'm hoping to get back to Mips soon. – Richard Pennington Jun 08 '15 at 00:34
  • That's pretty cool; we're using musl with gcc so far, musl+clang would seem to be the ideal combination (gcc is rather bloated imho) but we haven't found the time to set that up yet, there isn't much documentation (whereas `musl-cross` comes with an installer and prebuilt gcc cross-compilers..). – Thomas Jun 08 '15 at 02:19
  • If you're interested in prebuilt clang binaries as will as libraries for several targets, I have them at http://ellcc.org – Richard Pennington Jun 08 '15 at 02:24
  • Cheers; I'll take a look at ellcc.org later this week I think – Thomas Jun 08 '15 at 04:29
1

I don't speak MIPS assembly, but it doesn't look like you're setting up the "auxiliary vector" defined starting on page 3-30 of the MIPS ABI located here: http://www.linux-mips.org/pub/linux/mips/doc/ABI/mipsabi.pdf - this could be the cause of the problem?

Jules
  • 14,841
  • 9
  • 83
  • 130
  • I added the "auxiliary vector" on the stack but unfortunately it does not seem to make a difference :( – Thomas Jun 07 '15 at 09:33