It affects cold starts, always the perf characteristic that gets most attention in managed code because it is by far the slowest and most noticeable. Managed code just has a lot of DLLs to find, both assemblies and ngen-ed DLLs. That's slow on a hard drive, it takes a while to dig up the files when nothing is in the file system cache yet. Bigger directories take longer to search.
It is not an exclusive problem to managed code, native programs that use a lot of DLLs have this problem too. That's why big programs like Office apps or Acrobat Reader use "optimizers", a little program that gets started at log-in time that does nothing but touch the set of DLLs that the main program needs. Warming up the file system cache. And actually slowing down whatever program you really want to start when you login for the first time. I always delete them but they have a habit of getting put back, especially Adobe sucks that way. Windows Superfetch is the superior solution, it dynamically adjusts the set of executables to pre-cache, based on actual usage.
Actually removing assemblies from the GAC is of course not a realistic solution. The effect is small anyway.