Using IOC (inversion of control) rather than singletons initiated at first use is advantageous for other reasons too.
Singleton-initialisation can suffer (famously) from multi-threading issues where two threads try to access it for the first time simultaneously. Subsequent access is likely to be correctly synchonised but the first one is much harder to do.
Another huge advantage I have found of using IOC is when an error may occur in initialisation. You do not want this to happen on "first use", you want to know of this failure at an early stage, and certainly it is easier to handle the error this way.
Finally, with regards to testing, IOC provides the perfect model to isolate components, substitute them as necessary and bring together different combinations in a more flexible way, thus providing the perfect harness for both unit-testing and integration testing, as well as a good rollback mechanism without actually having to revert any code at all.
The general reason singletons are so often used is not for the one-ness but for the global-ness. If your project is managed correctly you have a single global object to which all the others "register" (thus your IOC model hangs off it) and are available globally whilst still being configurable.