Part of the issue with just concatenating the file isn't the time spent downloading, it's the time spent compiling on every page.
If you've got a 20,000 line file, and you only need 600 of those lines to get everything up and running (assuming that everything is written to be modular and asynchronous, using any pattern at all for managing the resources), then you're going to save yourself what might be half a second or more, if you serve the core program and extend as-needed (or on a delayed timer, serving large chunks of functionality which are closely related to one another).
The overall time spent downloading is higher.
The overall number of HTTP connections used is higher.
But the time required to make the page visible to the user is lower.
The time required to add base-functionality to the page is lower.
Then extra functionality can be streamed in, either after the load and initialization, or just-in-time, as-requested by the user, and in both cases, as long as the code you've streamed in is focused on doing that one thing, and isn't calling for a half-dozen other dependencies, the time between the request and the addition of the functionality is going to be minimal.
RequireJS uses a promise system, basically.
It allows you to declare dependencies up front, and hand in the code (as a callback) to be implemented, after all of its dependencies are handled.
If those dependencies have any dependencies, then they won't be initialized until their dependencies are loaded.
If you just want it loaded and order isn't important, then you don't need to give it dependencies.
The overall moral is, if you've got a system where all of your files are small, the overall weight of JS on the page is very small, you only needed a few hundred lines to do everything you wanted on the page... ...plus, you know where all of your dependencies are, you have a system on the server to make sure they're in the right order, et cetera (plus you've got great documentation, or you're the only one who touches this code, and you live inside of it, day to day)...
...then there's nothing at all wrong with doing what you're doing.
You might not see any difference at all, if the compile time is outweighed by the number of HTTP requests you make.
But for overall applications which are tens (or hundreds) of thousands of lines long, where only a fraction of that functionality is needed on any one page, there can be big savings in terms of perceived time between when the page loads, and when the app is "ready" for basic interaction by the user.