If the sub-projects exist solely to facilitate modular compilation (as you have mentioned in the comments on the question) then this attempt to "optimise" the build is at least partly responsible for causing the delays you are trying to eliminate.
When building in the IDE, the compiler will already only compile units that have changed since the previous compilation. Your sub-projects are forcing the compiler to make this assessment on the units involved at least twice. Once for each sub-project and then again for the "real" project.
In addition, the compiler is having to link and emit the EXE for each sub-project. EXE's which you say are not actually required as an output of your project build.
You will improve your build times by simply eliminating the redundant "sub-projects".
If the sub-projects added some other value, such as for example providing automated test coverage, then the overhead of these projects might be considered worthwhile, but this does not appear to be the case here.
One other thing to bear in mind is that the compiler decides whether to re-compile a unit based on whether each unit has itself changed. However, if conditional directives or other compiler settings have changed then all units should be recompiled otherwise the effect of the changed compiler settings is not applied (and can itself lead to strange behaviours and errors if these changed settings are applied to some units - the changed ones - and not others).
When building under the IDE this requires you to remember to perform a full build after making changes to compiler settings. In a dedicated build process it is generally advisable to always perform a full build in a "clean room" environment. That is deleting all compilation by-products (dcu's etc) either before or after each build.
NOTE: With these redundant sub-projects you again have created more work and potential trouble for yourself by having to ensure that your compiler settings are consistent across all sub-projects and the "master" project.