Nmap is an old and "grown" software package. It's better to start to read the excellent documentation and then the source instead the other way around.
I suggest Phrack 51 - The Art of Port Scanning and the slightly updated version as a still relevant primer.
Just to give you an glimpse on the mish mash nmaps internals are see this quote from the Phrack Articke above:
Prior to writing nmap, I spent a lot of time with other scanners exploring the
Internet and various private networks (note the avoidance of the "intranet"
buzzword). I have used many of the top scanners available today, including
strobe by Julian Assange, netcat by Hobbit, stcp by Uriel Maimon, pscan by
Pluvius, ident-scan by Dave Goldsmith, and the SATAN tcp/udp scanners by
Wietse Venema. These are all excellent scanners! In fact, I ended up hacking
most of them to support the best features of the others. Finally I decided
to write a whole new scanner, rather than rely on hacked versions of a dozen
different scanners in my /usr/local/sbin. While I wrote all the code, nmap
uses a lot of good ideas from its predecessors. I also incorporated some new
stuff like fragmentation scanning and options that were on my "wish list" for
other scanners.
(Disclaimer: haven't checked nmap's internals for a few years. Somebody might have cleaned up the code in between, but i consider it unlikely.