0

Recent Intel and AMD CPUs support specific AES instructions that increase the performance of encryption and decryption.

Is it possible to to detect when these instructions are called? For example by writing a kernel module that monitors the instructions that are sent to the CPU? Or is the kernel still to high-level?

arne.z
  • 3,242
  • 3
  • 24
  • 46

2 Answers2

1

My understanding is that instructions like AESENC require no special privileges, so you won't be able to trap them with one of the usual fault handlers even in the kernel. You might be able to do it with a modified version of QEMU or VirtualBox, but that would be a bit of a pain.

I'm guessing you are trying to see if a software package supports AES-NI?

David A
  • 344
  • 1
  • 4
  • My idea was to detect AES instructions in order to detect when files are being encrypted. But it seems like this is not feasible – arne.z May 04 '17 at 12:36
  • In that case, the only thing I can think of trying is using a debugger and using either single-stepping or perhaps reverse-engineering the binary itself using something like IDA Pro. Common encryption algorithms can often be recognized by their use of the AES-NI opcodes or by looking for the constants making up the [Rijndael_S-box](https://en.wikipedia.org/wiki/Rijndael_S-box). It would not be system-wide, though. – David A May 06 '17 at 19:28
0

The currently accepted answer is in some sense technically correct. It does not however answer your question.

Theoretically it would be possible to monitor when these instructions are used by any process. This would however create an infeasible amount of overhead. You would basically have to do a conditional branch every time any instruction is executed (!). It can be achieved for example by using an in-target probe, this is however a very costly and non scalable solution.

A more realistic but less precise solution is to do program counter sampling. You can check which instruction is being executed at a given moment by looking at a processes program counter. The program counter of a process can indeed be accessed from a kernel module. If you sample at a high enough resolution you can get a decent indicator of whether or not AES-NI instructions are being used.

rlf
  • 136
  • 2