This could simply be an artefact of the timing of the rebuilds vs the time you check. I am assuming that your daily rebuild is run, say, at 1am, but the performance test is not run until you are in at 8am. In this case, other scheduled tasks (especially other maintenance tasks) and general DB use can be run after the rebuild, but before the tests.
On the other hand, you probably test performance immediately after a manual rebuild. This results in different environments for your testing.
When SQL Server runs a rebuild, it will attempt to perform this in memory for enhanced speed. This will result in a large portion of the table (possibly all of it) being cached in memory, and accesses will be very quick at that point. However, as other processes work, parts of that table will be removed from memory to be used by other tables for caching their data. As more and more data is removed from memory for the first table, overall performance will degrade. Thus, testing after several hours can have significantly different results.
Also, overall load on the server can have a big effect on the testing. Even if the performance test is scheduled immediately after the automatic rebuild, if there are other processes running at that time (other index rebuilds, overnight data summarization, large reports, etc.) then performance may be significantly different from what you see with your manual rebuilds.
I have experienced the pattern you are seeing before, and it primarily came down to exactly what I talk about here - memory available for caching data. Performing some cache analysis to see what data is cached for each database can rule this situation out, as long as the cache analysis is run and recorded at intervals, so that you can see the caching over time.