So to restate your primary concern:
... and can therefore gain access to my application and its currently running forms/methods/controls/etc.
Before embarking on a complex and difficult means of loading, isolating, and restricting these extensions you should know a few things about Windows and the CLR. First, any program running on the box can use a number of Windows APIs to inject code into your process. Once code is loaded into your process, either by you or by the OS, accessing the CLR runtime and loading assemblies and/or running code in your existing AppDomain is fairly easy.
Knowing this you should weigh and balance the effort you excerpt to restrict 'extensions'. If I where building something like this I would be more concerned about other things than a malicious piece of extension code manipulating the state of my application. For example these are things you might consider:
- Only load extensions that are approved by your user, allow them to control what is allowed and allow them to revoke an extension later if desired. Look at Office or VStudio as an example.
- Ensure these approved extensions are untampered by using a code signing requirement (strong names, or code signing certificates).
- Consider having the ability to revoke an extension's ability to run remotely if it's found to be malicious.
- Prove a well-appointed Interface API to allow developers to easily implement desired behavior. If it's easy for them to use your interfaces to accomplish their task they won't need to 'hack'.
Beyond this there really isn't much else you can do. As I said, anyone can attack your application even with the above safe-guards. Your primary concern should be to not surprise your users. Thus the due-care in what code YOUR application runs is advisable, but what those extensions do once your users grant them access is not really something you can completely control.
This isn't to say that an AppDomain isolation won't provide you value, it may; however, IMHO getting the security restricted enough without limiting their ability to function will prove to be difficult.
UPDATE
... but if you load a plugin into an AppDomain which is configured with limited permissions how can it use this vector?
Correct, as I said in my closing statements, you can limit their access to unmanaged code within the AppDomain. This also limits their ability to develop a usable Windows experience. I expect that most WinForms applications use at least one PInvoke call or unmanaged COM control. This limitation imposed may be acceptable, I really can't say without more information about what functionality they are trying to provide.
All I was trying to say is that by installing and approving the extension your users are accepting the responsibility of allowing that extension to run. What that extension does and how malicious it may attempt to be is not your responsibility, assuming of course that you loaded the correct code. This is why I recommend focusing your energy on running approved code rather than worrying about what that code might do once it's in your process.