Illegal assembly instruction in C

Question

I am trying to get generate an assembly instruction trace using a C program. I am using asm functions to insert assembly instructions which are illegal in x86_64 and detect them.

I have the following C program -

int main()
{
    asm(".byte 0x16"); // Illegal in x86_64
    asm(".byte 0x17"); // Illegal in x86_64
    return 0;
}

This is the instruction trace for the main function, which is expected -

.text:004004d6 55                               push   %rbp
.text:004004d7 48 89 e5                         mov    %rsp,%rbp
.text:004004da 16                               (bad)  
.text:004004db 17                               (bad)  
.text:004004dc b8 00 00 00 00                   mov    $0x0,%eax
.text:004004e1 5d                               pop    %rbp
.text:004004e2 c3                               retq   
.text:004004e3 66 2e 0f 1f 84 00 00 00 00 00    nopw  %cs:0x0(%rax,%rax,1)
.text:004004ed 0f 1f 00                         nopl   (%rax)

I am using Intel PIN to run the program trace and based on the instruction I am getting I do some kind of analysis. So for the instructions which are (bad), they are illegal and thus cause the program to crash. I check the instruction address for every instruction in the trace along with the address of the instruction which is next.

The next instruction address for the instruction mov %rsp, %rbp (address 0x004004d7, the one before the 0x16 instruction) as shown by Pin is 0x0. 0x0 is shown by Pin when the next instruction is a conditional/unconditional branch instruction. Since it is shown for the bad instructions as well, I suspect that the bad instructions are calling an exception handler which terminates the program and gives the following message -

Illegal instruction (core dumped)

I am using Pin to instrument the instructions and when I get the illegal instructions, I do some kind of simulation and then delete this instruction so that the trace can proceed further.

Is there any kind of setting/flags to disable this illegal instruction exception generation?

I am using gcc to generate my executable and it simple compiles without generating an error. The problem comes when I execute the instruction. Is there any setting that simply just walks over the instruction, when it encounters an illegal instruction?

Why are you posting question after question on this subject? What is missing in your previous questions for you to ask another one? — fuz, Oct 09 '17 at 22:17
I'm not sure if it would work for you, but I would just catch the exception, patch the instruction to a `nop` and resume. (I can post example code for Linux if you like. But there's probably some pin-specific way to do it.) — David Schwartz, Oct 09 '17 at 22:18
@DavidSchwartz I am trying to do something similar, but it seems that the exception is raised before I am able to instrument the instruction. Yes, the linux example helps, please do share it — Rohit Poduri, Oct 09 '17 at 22:38
@fuz, I figured out all the things about the traces. It's just that I can't seem to figure out why the exception is raised as I am instrumenting the instruction, before it is executed. So according to my logic I should catch the instruction, then delete it so no exception should be raised. I am a bit new to Pin so I am not finding good documentation regarding what I want to do, so I found stackoverflow answers to be most helpful. I will try to reduce the number of questions asked in the future. — Rohit Poduri, Oct 09 '17 at 22:41

David Schwartz · Answer 1 · 2017-10-10T22:31:21.013

As requested, here is the code I used to patch a ud2 undefined instruction to nop nop on unicore machines and nop lock on multicore machines. I've removed code that's not at all relevant to your use case:

void __gmon_start__(void)
{
   if(ill) return;
   ill=true;
   struct sigaction act;
   memset((void *) &act, 0, sizeof(act));
   act.sa_sigaction=sigill;
   act.sa_flags=SA_SIGINFO;
   sigaction(SIGILL, &act, NULL);
   // more code to set IsUP goes here
}

And:

static void sigill(int, struct siginfo *info, void *extra)
{
    // Dynamically patch a 'ud2' to a 'nop lock' or a 'nop nop'

    // Begin with a full CPU sync (purge unmodified code)
    __asm__ __volatile__("cpuid" : : : "ax", "bx", "cx", "dx");

 // There was code here to acquire a mutex to prevent
 // multiple attempts to change the same code

  ucontext_t *uc=(ucontext_t *) extra;
  if (uc==NULL) { char *j=NULL; *j++; } // fatal

  unsigned char *instr=(unsigned char *) uc->uc_mcontext.gregs[14]; // EIP
  if(instr==NULL) { char *j=NULL; *j++; } // fatal


    if(instr[0] != 0x90)
    { // instruction wasn't already patched

     if ( (instr[0] != 0x0f) || // must be a ud2
            ((instr[1] != 0x0b) && (instr[1] != 0xb9)) )
      { char *j=NULL; *j++; } // fatal

      mprotect( (void *) ( ((int) instr) & ~(PAGESIZE-1)),
           PAGESIZE, PROT_WRITE | PROT_EXEC | PROT_READ);
      instr[0] = 0x90; // first byte is always a nop
      instr[1] = (IsUP) ? 0x90 : 0xf0;
      mprotect( (void *) ( ((int) instr) & ~(PAGESIZE-1)),
          PAGESIZE, PROT_EXEC | PROT_READ);
   }

  // mutex was released here

  // Reflush the CPU
  __asm__ __volatile__("cpuid" : : : "ax", "bx", "cx", "dx");
 }

Sorry the code is a mess.

Illegal assembly instruction in C

1 Answers1