The first call to CoCreateInstance
has to load into the process, and initialize, the .NET runtime. Then, your DLL has to be loaded, "verified", and compiled into machine code (although just-in-time helps a lot to speed-up the startup). The .NET runtime also has to parse the metadata of your assembly, and then dynamically generate and compile the "COM callable wrappers" (http://msdn.microsoft.com/en-us/library/f07c8z1c.aspx) which are the proxies that bridge between the unmanaged COM world and the managed .NET runtime. Any additional libraries your code might use also needs to be loaded, verified and possibly compiled into machine code (if not NGEN'd).
This is inherently an expensive process. The delays you mention are not unheard of.
I don't believe there is much you can do to speed things up. I suggest you think about whether you can take the hit early in your program's lifetime by creating an object soon after startup. It won't make it faster, but it might improve the user experience dramatically. if your program just can't tolerate the delays, then you should not use .NET to write the COM object (more specifically, you should not use .NET in your process at all. This is not an issue with using COM; it's an issue with loading .NET)
Incidentally, this is one of the reasons why writing shell extensions in .NET is... "highly discouraged". See this recent post on this subject, which touches on the startup performance of .NET as well: http://blogs.msdn.com/b/oldnewthing/archive/2013/02/22/10396079.aspx
(That's why I asked earlier what kind of client you were running. A client that already runs .NET managed code depends on the .NET runtime and would not be affected by these delays)