pointer casting in c [shellcode test]

Question

I'm following a tutorial on how to write a shellcode but I'm failing to understand what this pointer function casting is doing to the bytecode, could someone explain this to me?.

char code[] = "bytecode will go here!";
int main(int argc, char **argv)
{
   int (*func)();
   func = (int (*)()) code;
   (int)(*func)();
}

the code is something like this "\xb0\x01\x31\xdb\xcd\x80" for exiting it's the result of objdump of an assembly code that exits

Qubit · Answer 1 · 2018-07-23T08:18:55.280

0

You define a function pointer func that points to a function which will return an integer.

In the second line, you assign your bytecode to that function pointer, but you have to cast it to a function pointer first, there is no default conversion between a character array and a function pointer.

Lastly, you try to call the function that func points to, thereby hoping to execute your bytecode.

edited Jul 23 '18 at 08:18

answered Jul 23 '18 at 08:02

Qubit

1,175
9
18

It's worth noting that the las line's syntax is needlessly convoluted, a simple `func();` would be enough. – Quentin Jul 23 '18 at 08:05
thanks!! that clarified a lot of things. it makes sense now ^^ – Syched Flow Jul 23 '18 at 08:07
1

Note that is not a function pointer that takes no arguments. It takes arbitrary number of arguments! Just try! – Antti Haapala -- Слава Україні Jul 23 '18 at 08:17
Right, I've removed that line, thank you for the clarification. And yes, the last line is needlessly messy. – Qubit Jul 23 '18 at 08:19

score 0 · Accepted Answer · answered Jul 23 '18 at 08:17

The code itself has not only undefined behaviour, it is bad wherever it doesn't. Let's go through the lines:

Declare func as a pointer to a function returning an int, with no prototype.
```
int (*func)();
```
cast the array which decays to a pointer to the first character in the array, into a pointer to function returning int having no prototype, and assign it to func. This of course has totally undefined behaviour.
```
func = (int (*)()) code;
```
The third line is the worst of them all:
```
(int)(*func)();
```
Here we dereference the function pointer with *func, then it immediately decays back to a function pointer because it is subjected to the function call operator which only works with function pointers! Then the return value, which is an int, is cast to an int, and then discarded.

Of course the 3rd line should have been written as func();. And since the return value is very probably handled via registers, and we're not interested in it, the proto for the shellcode function should have been void shellcode(void). Lastly, you don't need a variable here for the function pointer. Therefore we could just write

( (void (*)(void)) code ) ();

i.e. cast the code as a pointer to function returning void taking no arguments, then call it.

pointer casting in c [shellcode test]

2 Answers2