3

The background: I've written a python script to inspect IP packets, specifically the payload/data of a packet in order to detect if it could be used in a buffer (stack) overflow. Now as I understand it a NOP sled is used to pad the stack so that the instruction pointer will eventually run into your exploit code, this I can easily detect by looking for repeating occurrences of 0x90. I've seen code with a lot of NOP commands to as few as 8 in the case of SQL slammer so I could perhaps use 8 as a minimum.

Now my question, are NOP sleds often used in legitimate code? If the answer is yes, are there a few specific cases (which means I can look for these cases and then rule out the packet as potentially harmless) or is this approach just not practical for identifying malicious code?

Trowalts
  • 119
  • 1
  • 2
  • 10

3 Answers3

9

The compiler will generate NOPs to align code -- for instance, on some iterations of the x86, jumps execute faster if the jump destination is aligned to a 4-, 8-, or even 16-byte boundary.

Some compilers try to use "long NOPs" when possible -- single instructions that take up more than one byte of space, and may formally do something, but have no effect on processor state -- as on some iterations of the x86 architecture this is faster. For instance, 66 90 is a two-byte NOP, and 8d 74 26 00 is a four-byte NOP (technically lea 0(%esi,%eiz,1),%esi, but as you can see that just copies the value in %esi to itself, so there's no effect). However, these can't be used in all cases, and the sequences that are fastest on some x86es are depressingly often really slow on others. I haven't read the current micro-optimization guidelines but I wouldn't be surprised if Intel and AMD were working to make a string of 90s the fastest way to do a long NOP, and had their compilers match.

zwol
  • 135,547
  • 38
  • 252
  • 361
  • 1
    Long `nop`s are still always better. The optimal long-NOP sequences differ for Intel and AMD, but multiple NOP instructions always go through the pipeline separately all the way to retirement. There's little need to optimize short-NOP handling because assemblers know how to use long NOPs, and compilers avoid putting them inside inner loops (at worst one long NOP inside an outer loop to align an inner loop). – Peter Cordes Nov 13 '17 at 06:10
3

from wikipedia:

A NOP is most commonly used for timing purposes, to force memory alignment, to prevent hazards, to occupy a branch delay slot, or as a place-holder to be replaced by active instructions later on in program development (or to replace removed instructions when refactoring would be problematic or time-consuming). In some cases, a NOP can have minor side effects; for example, on the Motorola 68000 series of processors, the NOP opcode will cause a synchronization of the pipeline.

Furthermore, 0x90 may conceivably be used by compilers as filler for uninitialized arrays, in case the data in the array gets interpreted as opcodes, it does nothing. You see a similar effect with Visual studio which fills uninitialized arrays with 0xCC which is int 3 that causes a breakpoint halt.

Further yet, any data in the executable may contain any number of 0x90 and it may not be trivial to differentiate between it and code.

shoosh
  • 76,898
  • 55
  • 205
  • 325
2

I just looked through the last binary I compiled (non-malicious x86 code on Linux), and found:

016b5e0 458b c9ec 90c3 9090 9090 9090 9090 9090

I think you can conclude that finding repeated sequences of 0x90 doesn't necessarily indicate malicious intent.

Greg Hewgill
  • 951,095
  • 183
  • 1,149
  • 1,285
  • C3 = RET, so that's aligning the next function to 16 bytes probably. I guess you could spot that case as something different. – Rup Oct 17 '11 at 00:27
  • 2
    Nothing says that compiled code always has to end with a RET (it could JMP somewhere else). – Greg Hewgill Oct 17 '11 at 00:29
  • Just in case any other noobs like me are wondering about @GregHewgill 's comment - - code like int funfunfunction(...){ for(...) { do things; if(something) return 3; do more things; } } would end in a jump instead of a return – Cedar Oct 15 '19 at 14:32