For Esxi
In ESXi vSphere client, you can restrict access to authorized IP.
In the Esxi server main panel > configuration > Software > Firewall > Incoming connexion
You need to restrict mainly:
- vSphere Client
- SSH (if you enabled it)
- vSphere Web Access
But like me you can restrict all execpt:
- DHCP client (if you use DHCP for ESXI IP)
- DHCPv6 (if you use DHCP v6 for ESXI)
- DNS Client (if you need the DNS client for ESXI)

For windows 2012
In you Windows 2012 go to Control Panel > System and Security > Windows Firewall > Advanced Settings
Add new inbound rules for port 3389 in TCP :

Then in scope tab to limit IP address add the IPs you want:
