Microsoft procedure for detecting open relays - manual telnet commands, too much to post up here. http://support.microsoft.com/kb/324958
Automated test http://www.mailradar.com/openrelay/ However when I've used this with Microsoft IIS6, the tests with the email address surrounded with speechmarks (e.g. method 7 RCPT TO: <"relaytest@mailradar.com">) appear to fail but don't actually get sent out. If you find yourself in this situation you may wish to do further investigating of settings and actually verifying these aren't ending up somewhere you don't want them to. 178.209.36.55 however sails through all these tests.
Once you are happy you are not running an open relay, the next step is to ensure you aren't backscattering. Backscatter is when a third party sends an email to an undeliverable address with a forged sender header so a NDR is sent to the apparent sender (forged).
This often comes about as a result of a server configured to relay mail for a whole domain without a list of valid users so it will accept the email initially, then will not be able to relay it to a server with a list of valid users.
Simple test - I'm going to call your external personal email address me@hotmail.com and your domain is domain.com
ehlo hotmail.com
mail from me@hotmail.com
rcpt to:invaliduser@domain.com
data
subject:subject
text
.
Then check the me@hotmail.com account (probably in the junk folder) for a NDR from something like postmaster@domain.com.