0

I'm doing x86-64 binary obfuscation research and fundamentally one of the key challenges in the offense / defense cat and mouse game of executing a known-bad program and detecting it (even when obfuscated) is system call sequence analysis.

Put simply, obfuscation is just achieving the same effects on the system through a different sequence of instructions and memory states in order to minimize observable analysis channels. But at the end of the day, you need to execute certain system calls in a certain order to achieve certain input / output behaviors for a program.

Or do you? The question I want to study is this: Could the intended outcome of some or all system calls be achieved through different system calls? Let's say system call D, when executed 3 times consecutively, with certain parameters can be heuristically attributed to malicious behavior. If system calls A, B, and C could be found to achieve the same effect (perhaps in addition to other side-effects) desired from system call D, then it would be possible to evade kernel hooks designed to trace and heuristically analyze system call sequences.

To determine how often this system call outcome overlap exists in a given OS, I don't want to use documentation and manual analysis for a few reasons:

  • undocumented behavior
  • lots of work, repeated for every OS and even different versions

So rather, I'm interested in performing black-box analysis to fuzz system calls with various arguments and observing the effects. My problem is I'm not sure how to measure the effects. Once I execute a system call, what mechanism could I use to observe exactly which changes result from it? Is there any reliable way, aside from completely iterating over entire forensic snapshots of the machine before and after?

J.Todd
  • 707
  • 1
  • 12
  • 34
  • In terms of true and intended process state, there's stuff like `/proc/PID/maps` (memory mappings) and `/proc/PID/fd` (set of open file descriptors, and what files they're open on, and `/proc/PID/fdinfo` - file positions). Those are Linux paths; other OSes may not expose as much. But in terms of possible exploits of kernel vulns, internal details might matter that don't usually get reported via normal mechanisms. e.g. maybe one populates some kind of cache with one thing vs. another. Also, things like registering signal handlers may be harder, not reported in a standard way. – Peter Cordes Feb 04 '22 at 19:39
  • (That's just my first thought; IDK if it's really an answer.) – Peter Cordes Feb 04 '22 at 19:40
  • @PeterCordes So I'm thinking it's really hard because like you say, a lot of the system calls might put things in places in kernel memory space that we can't see without using a driver, which even after going through all that trouble you would have to run every system call / argument combination many times to weed out which changes can actually be attributed to the system call vs some other activity on the system. – J.Todd Feb 04 '22 at 20:06
  • @PeterCordes and I also suspect my theory of being able to replace system calls is bad. Let's say there was enough overlap to do it consistently, the approach would still be a poor form of obfuscation, I now realize: Since userland programs almost exclusively achieve system calls not directly, but by jumping to shared memory subroutines abstracting them (OS APIs). Which means if we deviate from system call patterns consistent with the behavior of those abstraction layers and execute syscalls in arbitrary order, an endpoint security product tracing the activity could almost certainly notice. – J.Todd Feb 04 '22 at 20:10
  • Are you thinking about Windows and WinAPI DLLs? It's common on POSIX OSes for programs to use `open`, `read`, `select`/`poll` and `mmap` directly, and the libc wrappers for those are just thin wrappers around one true syscall each. And the BSD socket API is similarly plain system calls, like `listen(2)` / `accept(2)`. There might be some kinds of patterns that are unusual; if a system normally runs only a few server programs then I guess machine learning might be able to learn its usual patterns? IDK, I've never looked into pattern analysis security stuff. – Peter Cordes Feb 04 '22 at 20:41
  • @PeterCordes Yes I was referring to WinAPI and things like standard c lib. "if a system normally runs only a few server programs then I guess machine learning might be able to learn its usual patterns?" My thinking is more along the lines of recording the sequences per-process rather than as a global list. And ah, I thought each API call did multiple syscalls. I will have to look at WinAPI under a debugger some time. – J.Todd Feb 04 '22 at 21:14

0 Answers0