Static executable segfaults if location counter is initialized as too small or too large in linker script

Question

I'm trying to generate a static executable for this program (with musl):

main.S:

.section .text
    .global main

main:
    mov $msg, %rdi
    mov $0, %rax
    call printf

    mov %rax, %rdi
    mov $60, %rax
    syscall

msg:
    .ascii "hello world from printf\n\0"

Compilation command:

clang -g -c main.S -o main.o

Linking command (musl libc is placed in musl directory (version 1.2.1)):

ld main.o musl/crt1.o -o sm -Tstatic.ld -static -lc -lm -Lmusl

Linker script (static.ld):

ENTRY(_start)
SECTIONS
{
    . = 0x100e8;
}

This config results in a working executable, but if I change the location counter offset to 0x10000 or 0x20000, the resulting executable crashes during startup with a segfault. On debugging I found that musl initialization code tries to read the program headers (location received in aux vector), and for some reason the memory address of program header as given by aux vector is unmapped in our address space.

What is the cause of this behavior? What exactly is the counter offset in a linker script? How does it affect the linker output other than altering the load address?

Note: The segfault occurs when the the musl initialization code tries to access program headers

Answer 1

There are a few issues here.

1. Your main.S has a stack alignment bug: on x86_64, you must realign the stack to 16-byte boundary before calling any other function (you can assume 8-byte alignment on entry).
   Without this, I get a crash inside printf due to movaps %xmm0,0x40(%rsp) with misaligned $rsp.

2. Your link order is wrong: crt1.o should be linked before main.o

3. When you don't leave SIZEOF_HEADERS == 0xe8 space before starting your .text section, you are leaving it up to the linker to put program headers elsewhere, and it does. The trouble is: musl (and a lot of other code) assumes that the file header and program headers are mapped in (but the ELF format doesn't require this). So they crash.

The right way to specify start address:

ENTRY(_start)
SECTIONS
{
    . = 0x10000 + SIZEOF_HEADERS;
}

Update:

Why does the order matter?

Linkers (in general) will assemble initializers (constructors) left to right. When you call standard C library routines from main(), you expect the standard library to have initialized itself before main() was called. Code in crt1.o is responsible for performing such initialization.

If you link in the wrong order: crt1.o after main.o, construction may not happen correctly. Whether you'll be able to observe this depends on implementation details of the standard library, and exactly what parts of it you are using. So your binary may appear to work correctly. But it is still better to link objects in the correct order.

I'm leaving 0x10000 space, isn't it enough for headers?

You are interfering with the built-in default linker script, and instead giving it incomplete specification of how to lay out your program in memory. When you do so, you need to know how the linker will react. Different linkers will react differently.

The binutils ld reacts by not emitting a LOAD segment covering program headers. The ld.lld reacts differently -- it actually moves .text past program headers.

The resulting binaries still crash though, because the binary layout is not what the kernel expects, and the kernel-supplied AT_PHDR address in the aux vector is wrong.

It looks like the kernel expects the first LOAD segment to be the one which contains program headers. Arguably that is a bug in the kernel -- nothing in the ELF spec requires this. But all normal binaries do have program headers in the first LOAD segment, so you'll just have to do the same (or convince kernel developers to add code to handle your weird binary layout).