0

I tried to write a self-modifying code. (Refer to the link https://shanetully.com/2013/12/writing-a-self-mutating-x86_64-c-program/) The self-modifying code works when there is no optimization (-o0)

gcc -O0 smc.c -o smc

Calling foo...
i: 1
Calling foo...
i: 42

while when the optimization level increases (-O1-O2-O3..) Self-modifying code no longer works.

gcc -O3 smc.c -o smc

Calling foo...
i: 1
Calling foo...
i: 1

Is it possible to make self-modifying code work with -O3 level optimization, and what should I do?

The program is as follows:

#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/mman.h>

void foo(void);
int change_page_permissions_of_address(void *addr);

int main(void) {
    void *foo_addr = (void*)foo;

    // Change the permissions of the page that contains foo() to read, write, and execute
    // This assumes that foo() is fully contained by a single page
    if(change_page_permissions_of_address(foo_addr) == -1) {
        fprintf(stderr, "Error while changing page permissions of foo(): %s\n", strerror(errno));
        return 1;
    }

    // Call the unmodified foo()
    puts("Calling foo...");
    foo();

    // Change the immediate value in the addl instruction in foo() to 42
    unsigned char *instruction = (unsigned char*)foo_addr + 22;
    // Notice that 22 here is the offset that I compiled. Different compilations and machine offsets may vary.
    *instruction = 0x2A;

    // Call the modified foo()
    puts("Calling foo...");
    foo();

    return 0;
}

void foo(void) {
    int i=0;
    i++;
    printf("i: %d\n", i);
}

int change_page_permissions_of_address(void *addr) {
    // Move the pointer to the page boundary
    int page_size = getpagesize();
    addr -= (unsigned long)addr % page_size;

    if(mprotect(addr, page_size, PROT_READ | PROT_WRITE | PROT_EXEC) == -1) 
    {
        return -1;
    }

    return 0;
}
gazile
  • 105
  • 6
  • 1
    Provide a [mcve], please! Also, did you look at the generated code? Self-modifying code is inherently brittle and depends a lot on the compiler's output. – Ulrich Eckhardt Mar 10 '22 at 06:16
  • I updated an example in the original post. I looked at the generated code and analyzed the offset of the bytes to be modified from the assembly code. – gazile Mar 10 '22 at 06:51
  • Which assembly code? The -O changes what is generated. Also, it's likely the function gets inlined. – Ulrich Eckhardt Mar 10 '22 at 07:39
  • Thanks so much! I solved it! The problem is that you say inline, because optimization is turned on, the function is expanded inline, so it is invalid. I declare __attribute__((noinline)) in front of the function that needs to be self-modified. – gazile Mar 10 '22 at 08:01

0 Answers0