You cannot just collect unused blocks and deal them out arbitrarily. Say, some site is allocated 120.0.0.0/8 and it turns out that half of it is not used, say, 120.128.0.0/9. There is a route to 120.0.0.0/8 announced through BGP. Now if you split the block, you need most likely two routes. Doesn't look exciting, but the routing tables in the Default-Free Zone (a.k.a. the Internet backbone) hold already way more than half a million entries. If you double a substantial number of routes, then the number of entries will grow considerably.
Now suppose the more likely case that the site has spread its addresses over the entire net block 120.0.0.0/8. Say, every other /16 is being populated, e.g. 120.1.0.0/16, 120.3.0.0/16, etc. If the site returns the unused "class B" blocks, and the responsible RIR deals them out to other sites, the number of routing table entries is increased by 255. If you do that for many sites, the number of routing table entries is going to explode, and this will be a serious bottleneck.
In practice the "average" scenario will probably be somewhere in the middle of the two scenarios, but as a matter of fact, there would be an awful lot of new "class C" networks that need separate routing table entries.
So why not simply renumbering networks and "defragmenting" the address space? Because renumbering means considerable downtime, since with IPv4 renumbering means in most cases to manually touch (or even reboot) each affected machine. And this is not taking into account all the hardcoded IP-addresses, which means you have to find an fix all affected applications. So from a practical point of view renumbering is next to impossible, and thus any plan to reclaim unused space will meet strong resistance.