IP of ubuntu server is accessible even after it listens on domain

Question

So my problem is that my site is accessible also with IP like https://xx.xx.xx.xxx something that I don't want. I only need domain name accessing servers IP. I tried virtual hosts on apache2 to redirect to domain but I think I can't make it after many tries like

<VirtualHost *:443>
ServerName test.com
DocumentRoot /var/www/html

SSLEngine on
SSLCertificateFile /path/to/your/ssl/certificate.crt
SSLCertificateKeyFile /path/to/your/ssl/private/key.key
SSLCACertificateFile /path/to/your/ssl/ca/certificate.crt