There is no simple solution such as an API you can call to restrict a process or a thread.
What IE does in protected mode (on Vista and Windows 7) is to load plug-ins into a separate process at a low integrity level. Processes running in low integrity mode have less access to system resources and are more isolated from higher integrity level processes. You can also set up ACLs on things like file system objects and registry keys that control whether or not low-integrity processes can access them. This limits the amount of damage they can do. This is a form of sandboxing or (depending on how strictly you define it) virtualization.
It's a lot of work to get it right. A low-integrity process can be so restricted that it needs help to do much of anything. When IE launches a protected mode process, it gives it a channel for communicating back to the main IE process. The plug-in can then make requests via this channel to do things like make registry changes and write to the file system. IE considers the requests and if it determines that it should be allowed, the IE process will do it on behalf of the plugin.
Another approach (which can be used to complement the sandboxing) is to require the DLL to be signed with a valid certificate. The signing allows you to have a little more trust in the DLL, because the certificate identifies a responsible party. The signature also ensures that nobody has tampered with the DLL.
Yet another approach is to hook all of the OS API calls you want to restrict. There are libraries for this, but the technique relies on a bit of OS hackery that could break on any update. The idea would be to create a process with some of your code that sets up a hook for each API you want to restrict and then loads the untrusted code and executes it. If the DLL calls a hooked API (e.g., CreateFile
), you're code will get called instead, and it can decide whether to return an error or pass the call onto the real OS API. This is difficult because the number of APIs you have to hook can be huge. Also, if the DLL knows that this is the technique in use, there are things it can do to subvert it.
Finally, you can do true sandboxing by not running native code at all. Instead of loading an untrusted DLL, you load code that's been compiled into some intermediate form and interpret it. This gives your interpretter complete control over what the program can do. This is also hard to implement, and it greatly lowers performance.
These are the extremes you have to go through if you're going to run untrusted code.