Dnsmasq not resolving local domains with different port number

Question

I have my dnsmasq config setup like:

no-resolv
address=/domain1.test/127.0.0.1
address=/domain2.test/127.0.0.1
port=53

And my system fails to resolve domain1.test but, domain2.test works fine. The only difference is domain2.test is a HTTP server that runs on port 80 and domain1.test is also a HTTP server but, it runs on port 81.

How can I resolve domain1.test locally to 127.0.0.1:81?