I believe way hyperthreading works is that when running a single thread, on average only half the available stages in the core's pipeline are in use. HT tries to "interweave" two threads to improve CPU utilisation.
This does mean each of the two threads can run slower than it otherwise would since they have to wait for one another, while the overall throughput of a given core is improved. However, if you're not running under load, I'd say it's unlikely that the "peanut gallery" will get scheduled on the core that is busy as opposed to the cores that aren't doing anything.
(Of course, this being modern CISC performance, it's hard to say anything with certainty based on a theorethical analysis.)