Here is a file that has a description of M4A (best I could find so far) on page 67:
http://iweb.dl.sourceforge.net/project/audiotools/audio%20formats%20reference/2.14/audioformats_2.14_letter.pdf
A typical M4A begins with an 'ftyp' atom indicating its file type...
10.2.1 the ftyp atom
[0 31] ftyp Length [32 63] 'ftyp' (0x66747970)
[64 95] Major Brand [96 127] Major Brand Version
[128 159] Compatible Brand₁ ...
The 'Major Brand' and 'Compatible Brand' elds are ASCII strings.
'Major Brand Version' is an integer.
At first I figured 'ftyp' would be where format is determined, but judging by this list that is more like the file type itself (already known as m4a):
http://www.ftyps.com/index.html
http://www.ftyps.com/what.html Describes a bit more of the format.
If ftyp doesn't differentiate, then I think that the 'Major Brand' field might refer to the fourcc's on this page:
http://wiki.multimedia.cx/index.php?title=QuickTime_container
The one for Apple Lossless being 'alac' and AAC is probably 'mp4a'
Apple's Lossless format open source page indicates that the ftype is 'alac' (slightly contradictory to above)
http://alac.macosforge.org/trac/browser/trunk/ALACMagicCookieDescription.txt
So far what I can tell is that the 4 bytes following ftyp are always (in a smallish sample size) 'M4A '.
Somewhere in the first ~200 (hex) bytes or so there is an ascii 'mp4a' for AAC compression or an 'alac' for Apple Lossless. The 'alac' always seems to come in pairs ~30 bytes apart ('mp4a' only once).
Sorry that's not more specific, if I find the exact location or prefix I'll update again. (My guess is the earlier part of the header has a size specified somewhere.)