1) What is the purpose of using certificates while opting for Message
security with client credential of UserName?
The X509 Certificate creates a secure channel - it is used to encrypt and digitally sign messages. Additionally, it authenticates the service to the client.
2) Is it not possible to use Message security with UserName client credential
without using a certificate?
Since the client's credential is just a username\ password pair, you need some sort of mechanism to create the secure channel. However, you can get around this with a custom binding. See this link for such a binding.
3) Can I use a self-signed certificate in production?
You certainly can use them. Whether you should or not is something up to debate. It depends on your situation. For example, if you're doing some form of Ecommerce then you definitely should use CA Signed Certificates.