The information that you have provided is a bit sparse. You have listed the following as one group of code:
<?xul-overlay href="chrome://messenger/content/messenger.xul"?>
<?xul-overlay href="chrome://messenger/content/messengercompose/messengercompose.xul"?>
From this, it is not clear if you are attempting to overlay two separate .xul
files from within the same overlay file.
Overlays are specified with a one-to-one correspondence between the file being overlayed and the overlay file. In other words, to overlay those two files, you need two separate files to have two overlays (or perhaps you could overlay the same information from one file onto two).
Overlays are defined in the chrome.manifest file. Use a single line per file you are overlaying. The syntax looks like:
overlay chrome://URI-to-be-overlaid chrome://overlay-URI [flags]
The [Flags] are described on MDN.