1

I am writing a Pin tool where I want to detect an instruction with a particular opcode. I have an executable from a sample C program where I am printing hello world. The architecture used is x86_64 and I have the trace of assembly instructions in the program. I am giving this program's assembly instruction trace to the Intel PIN tool, on which I am running the instruction trace and performing operations.

My goal here is to detect a particular opcode from my original program. For example I have added the following line in my C program-

asm(".byte 0x17");   // pop ss in 32-bit mode, but illegal in x86-64

Due to this 0x17 is an instruction in my C program trace. In my Intel PIN tool I have an instruction trace like this-

VOID Instruction(INS ins, VOID *v) 
{
   if(INS_Opcode(ins)==0x17)
   {
        //Do something
   }
}

However when I run my instruction trace, my Pin tool fails due to the illegal instruction 0x17 I have placed.

When I print the instruction opcodes as they come, I see that they have different opcodes from the x86_64 ones. For example the instruction AND has opcode 0x17 when I print. Do I have to do some kind of decoding or I am I detecting the opcodes in the wrong way?

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
Rohit Poduri
  • 99
  • 2
  • 9
  • This is a re-ask of https://stackoverflow.com/questions/46568487/emulation-of-new-instruction-in-intel-pin. Either this is a duplicate of that, or vice versa. – Peter Cordes Oct 04 '17 at 19:25
  • Does AND decode "wrong" even before the `0x17` is reached? How is PIN supposed to know it's a single-byte instruction, rather than the start of a longer instruction it doesn't recognize? Custom decoding also requires instruction-length decoding. (I haven't used PIN, so IDK how to do that.) – Peter Cordes Oct 04 '17 at 19:36
  • The pin opcode is not assembly opcode. They are constants from this file: https://software.intel.com/sites/landingpage/xed/ref-manual/html/xed-iclass-enum_8h.html. There is also invalid opcode, it might work as a trigger for your custom decoding. – rkapl Oct 04 '17 at 20:06
  • Thank you very much @rkapl. Now I understand why I was decoding the wrong instruction. Is there any way I can get the correct assembly opcode as I need to detect the 0x17 instruction and then perform some task based on it. – Rohit Poduri Oct 04 '17 at 20:45
  • @rkapl I tried using using the XED_ICLASS_INVALID enumarator, but my programs fails to recognize this and crashes due to the illegal instruction. – Rohit Poduri Oct 04 '17 at 21:43
  • Print all the instruction opcodes as they come, this will be be quicker then guessing. – nitzanms Oct 05 '17 at 18:15
  • @nitzanms I can't do that as the instructions will be executed as they come. For the illegal instruction, I would be modifying it during run time so that the original illegal instruction is not executed and the program can proceed. I want to know of a method to detect the illegal instruction and send it to another anlysis routine, where I will be emulating something else. – Rohit Poduri Oct 06 '17 at 01:10
  • What I'm suggesting is that you print information about the instruction to know how the illegal instruction looks like, and then write your pintool to accommodate pin's behavior. – nitzanms Oct 08 '17 at 08:29

0 Answers0