Based on other topics I know that Asterisk 1.8.* supports MD5 as the protocol for authentication
I want to know if the Astrisk 1.8.* support the key exchange RSA and DSS protocols with sip clients/servers?
If you mean are referring to the SIP challenge/response authentication method then RSA and DHS (I'm guessing you meant Diffie-Hellman DHS rather than DSS) aren't applicable since what's required is a cryptographic hashing algorithm such as MD5 or SHA2 rather than an asymmetric encryption algorithm (such as RSA) or key exchange sequence (such as DHS). Note that currently SIP only supports MD5 but it looks like there is a new RFC to move to SHA2.
If you are looking for confidential SIP transmissions then SIP does have support for that by way of TLS. The SIP RFC does mandate the support of a secure transmission channel which uses a sips prefix for SIP addresses instead of the standard sip and indicates that the SIP messages should be sent over a TLS channel. Not that while the SIP RFC mandates support for the TLS channel in practise it's not widely supported amongst public SIP services. Asterisk does support SIP TLS.