Here are the first things I would consider to solve this:
Large lists of options in work-item fields/drop-downs, especially lists of users. Have a big company? By default your Assigned To field will show all users who have access to the team projects. This could be a VERY large list that will take time to load and work with in Excel. Solution: Limit the users shown in the drop-down to only the ones required. I found this blog post that has the steps for you.
Anti-virus? I've found anti-virus to be an issue with some clients I have; more specifically, real-time scanning of files. I'm not sure how it works with the Excel publishing but there are various caches that could be effected during getting and refreshing or publishing of data depending on what you are doing. You can look at excluding the cache directories from live/real-time scanning of the anti-virus if this is the case. This steps to perform this vary depending on the system you use and how it is managed (centrally via policy or locally). There location of the cache on the client side can vary depending on the OS being used but you can look for the work item cache under the user profile directory or more likely under the hidden Program Data folder in windows (sometimes clearing this cache by deleting the files in this dir can solve problems too, but I'm assuming your behavior is the same on different compters so that probably won't help you)... Example of cache location is:
C:\Users\username(Team Explorer user name)\AppData\Local\Microsoft\Team Foundation\3.0\Cache(os: Windows Server 2008 R2)
As for the server cache directories there default location for the web access cache is:
C:\ProgramData\Microsoft\Team Foundation\Web Access\Cache_v10.0.30319
SQL Server Performance. If other operations are slow you may want to check the performance of your SQL Server hosting the TFS data.
Tracing TFS. Other than that you can turn on tracing on the TFS server and see more detail about what is going on.