Raising BRK in assembly on i386 Linux

Question

I found and studied x86 memory access segmentation fault and it won't work in my code. The difference perhaps being that I don't use separate .text and .data segments but keep all in a single segment by creating a custom ELF header. Does that explain why the SYS_BRK call fails?

The program then continues by making the memory pages read/write/execute etc. I tried to find the minimum code sample that illustrated the issue.

In kdbg the sample does work, but not when started from the command line, hence the message printing.

            cpu   386
            bits  32

; System calls used
%assign     SYS_EXIT    1
%assign     SYS_WRITE   4
%assign     SYS_BRK     45
%assign     SYS_MPROTECT 125

; flags for SYS_MPROTECT
%assign     PROT_READ   1
%assign     PROT_WRITE  2
%assign     PROT_EXEC   4

%assign     STDOUT      1

memstart:   org         0x08048000

ehdr:                                           ; Elf32_Ehdr (see https://en.wikipedia.org/wiki/Executable_and_Linkable_Format)
            db      0x7F, "ELF"                 ; e_ident[EI_MAG0..EI_MAG03]
            db      1                           ; e_ident[EI_CLASS]
            db      1                           ; e_ident[EI_DATA]
            db      1                           ; e_ident[EI_VERSION]
            db      0                           ; e_ident[EI_OSABI]
            db      0                           ; e_ident[EI_ABIVERSION]
  times 7   db      0                           ; e_ident[EI_PAD]
            dw      2                           ; e_type
            dw      3                           ; e_machine
            dd      1                           ; e_version
            dd      start                       ; e_entry
            dd      phdr - $$                   ; e_phoff
            dd      0                           ; e_shoff
            dd      0                           ; e_flags
            dw      ehdrsize                    ; e_ehsize
            dw      phdrsize                    ; e_phentsize
            dw      1                           ; e_phnum
            dw      0                           ; e_shentsize
            dw      0                           ; e_shnum
            dw      0                           ; e_shstrndx
ehdrsize    equ     $ - ehdr

phdr:                                           ; Elf32_Phdr
            dd      1                           ; p_type
            dd      0                           ; p_offset
            dd      $$                          ; p_vaddr
            dd      $$                          ; p_paddr
            dd      filesize                    ; p_filesz
            dd      filesize                    ; p_memsz
            dd      5                           ; p_flags
            dd      0x1000                      ; p_align
phdrsize    equ     $ - phdr

memsize:    dd      16*4096                     ; = 16 * 4K pages
memtop:     dd      0


start:      ;int     3
            xor     ebx, ebx                    ; find the amount of allocated memory
            mov     eax, SYS_BRK
            int     0x80                        ; eax contains current memtop

            sub     eax, memstart               ; got enough memory?
            test    eax, [memsize]
            ja      memgood

            mov     eax, memstart               ; raise memory limit to memstart + memsize
            add     eax, [memsize]
            mov     ebx, eax
            mov     ecx, eax                    ; save requested memory size in ecx
            mov     eax, SYS_BRK
            int     0x80
            cmp     eax, ecx
            jne     brk_error                   ; raising memory limit failed

memgood:    mov     edx, (PROT_READ | PROT_WRITE | PROT_EXEC)
            mov     ecx, [memsize]              ; make memory read/write/execute
            mov     ebx, memstart
            mov     eax, SYS_MPROTECT
            int     0x80
            test    eax, eax
            js      bailout

            jmp     launch                      ; lets start the party


brk_error:  mov     edx, brkelen
            mov     ecx, brke
            mov     ebx, STDOUT
            mov     eax, SYS_WRITE
            int     0x80
            jmp     bailout
brke:       db      'SYS_BRK failed, bye', 10
brkelen     equ     $ - brke

bailout:    mov     eax, SYS_EXIT
            xor     ebx, ebx
            int     0x80

launch:     mov     edx, succlen
            mov     ecx, succ
            mov     ebx, STDOUT
            mov     eax, SYS_WRITE
            int     0x80
            jmp     bailout
succ:       db      'Success with mem config, bye', 10
succlen     equ     $ - succ

filesize    equ $ - $$

I saw your answer and verified it worked. The header is regular except that there is a single segment, so no code/data/bss separation. (oops, enter suddenly posts). I edit the question to show the complete initialisation, but that didn't make a difference so I left those parts out. — LNoor, Apr 03 '16 at 15:19
Sorry, (first time user). The problem with my code is the second call to BRK. When I put an `int 3` in front of the `int 128` I get the expected msg about a debugger breakpoint. If I put it after the program never gets there. When run from the console the program appears to just hang. — LNoor, Apr 03 '16 at 15:33
I've removed all my comments because you now provided the actual code lol — Michael Petch, Apr 03 '16 at 15:33
I haven't finished understanding your code yet, but mixing code and data makes inefficient use of L1 cache. Intel and AMD CPUs use split L1I and L1D caches, so for example loading the `brke` string will require that cache line to be in L1D cache, while the surrounding code will require the cache line to be in L1I cache. OTOH, it does mean the line is hot in L2. Also, you make code fetching less efficient by having to `jmp` over data. Normally you put strings and other read-only data in `.rodata`, which groups them together separate from code, but still in the `.text` section. — Peter Cordes, Apr 03 '16 at 20:34

Timothy Baldwin · Answer 1 · 2016-04-03T21:49:49.637

You should use cmp here instead of test:

    sub     eax, memstart               ; got enough memory?
    test    eax, [memsize]
    ja      memgood

The memory area described by SYS_BRK start a random offset of 0 to 0x02000000 after the executable unless address space layout randomisation is disabled, which I suspect your debugger does. You can use mmap to allocate memory at a specified address (don't set MAP_FIXED unless you want overwrite existing mappings) .

However this whole exercise with brk and mprotect seems rather pointless as apart from the stack upon program start memory is allocated exactly as the ELF header specified, instead you can:

phdr:                                           ; Elf32_Phdr
            dd      1                           ; p_type
            dd      0                           ; p_offset
            dd      $$                          ; p_vaddr
            dd      $$                          ; p_paddr
            dd      filesize                    ; p_filesz
            dd      16*4096                     ; p_memsz
            dd      7                           ; p_flags
            dd      0x1000                      ; p_align
phdrsize    equ     $ - phdr

Thank you Timothy. That is probably the way to go. The program sort of rewrites and enlarges itself claiming the extra space to do so. I didn't consider address space layout randomization. But modifying the Elf header and relaunching should do the trick. — LNoor, Apr 04 '16 at 20:22
Disabling address space randomization proved to be the solution. So the program is started as `setarch $(uname -m) -RL ./lf` which solved the problem. Thanks @Timothy. See [http://stackoverflow.com/questions/5194666/disable-randomization-of-memory-addresses]. — LNoor, Apr 05 '16 at 19:54

Raising BRK in assembly on i386 Linux

1 Answers1