Assuming you're talking about HTTPS, the rules traditionally used for host name verification are defined in RFC 2818, Section 3.1, which doesn't mention Internationalized Domain Names at all.
More recently, a "best practice" RFC, RFC 6125, was written to harmonise the host name verification procedures across protocols and clarify a number of other points. Here is what it says about IDN (section 6.4.2):
If the DNS domain name portion of a reference identifier is an
internationalized domain name, then an implementation MUST convert
any U-labels [IDNA-DEFS] in the domain name to A-labels before
checking the domain name. In accordance with [IDNA-PROTO], A-labels
MUST be compared as case-insensitive ASCII. Each label MUST match in
order for the domain names to be considered to match, except as
supplemented by the rule about checking of wildcard labels
(Section 6.4.3; but see also Section 7.2 regarding wildcards in
internationalized domain names).
Unfortunately, this may not help you in practice. Firstly, RFC 6125 is relatively recent, and few applications or libraries claim to implement it as far as I know. Secondly, not all libraries follow RFC 2818 to the letter anyway (browsers can sometimes be more tolerant regarding which CNs are acceptable, for example).