5

I am currently experimenting a lot with java's security mechanisms in order to understand how to best execute untrusted code in a sandbox. One of the things you want to protect against are infinite loops which is why ideally you want to run the untrusted code in its own thread. Now, of course, malicious code could, for example, do some heavy processing resulting in a hanging thread. To get rid of this thread essentially the only way is to use java's deprecated Thread.stop() mechanism. The main reason this is problematic is that all of the locks held by the thread are released which could result in corrupted objects.

The question is: with Java's SecurityManager and custom classloaders I am able to track, for example, what classes can be loaded and what system resources can be accessed. Is there a way to be informed (and to potentially prohibit) code from acquiring locks (for example, defining a callback that is informed before the current thread goes into a synchronized block).

Arno Mittelbach
  • 952
  • 5
  • 12

1 Answers1

4

If you are already using a custom classloader, you could inspect the bytecode of each class before loading it and detect it if contains an instruction that grabs a lock (monitorenter).

You should also consider that locks released with stop() are only a problem if they are acquired on shared objects that other code might potentially lock. If you can avoid such objects to be accessible in the "evil" thread, you're safe.

Alessio Stalla
  • 1,022
  • 6
  • 22
  • True, although ideally I do not want to discard classes just because they potentially acquire locks. I could of course change the bytecode while loading and insert callbacks except that this does not work for any class in java.*. Is monitorenter the only bytecode instruction that tells the vm to grab a lock? – Arno Mittelbach Jun 07 '13 at 09:20
  • Yes, however, monitorenter is only emitted when compiling synchronized blocks, whereas when calling synchronized methods the lock acquisition is implicit. So you would also have to prevent the thread to obtain access to shared objects with synchronized methods. If the code is in java.*, though, you should assume that it's not evil :) – Alessio Stalla Jun 07 '13 at 12:40
  • The code in java.* is not evil. However, if an object from java.* is shared for example a queue from java.concurrent then the lock is acquired from within java.* even though the object is used from the outside. – Arno Mittelbach Jun 07 '13 at 12:50
  • Right - what I mean is that you'd have to check user code calling methods that potentially lock, not library code that happens to use locks. monitorenter in java.* is fine, Vector.add() on a shared vector in com.evil.hack.* is not fine. – Alessio Stalla Jun 07 '13 at 13:13