0

Using ESXi v6 with the following setup:

One standard Switch: vSwitch0

  • vSwitch0 has one uplink physical adapter (Internet connected, and this is the only physical adapter)
  • vSwitch0 VM Network has 3 virtual machines (Web Accessible, using Static IPs)

I would like to add a 4th VM running PfSense, through which all the Internet traffic inwards and outwards will go through, to the 3 VMs I've already got up and running.

Something along the lines of the 3 VMs having an internal private network (say in the 10.10.10.0 range), using the pfSense firewall as the Gateway. One way this might work is if I created a LAN only Network. Each of the 3 VM NICs will be connected to this LAN only Network (using 10.10.10.100 as Gateway suppose).

PfSense will have two NICs, one connected to the Physical NIC and one connected to the LAN only network, with the IP address 10.10.10.100.

What I've already done:

  • Created a new vSwitch, not connected to any physical NIC
  • Assigned the 3 VM NICs to this network

Do I need to set the gateway for these VM NICs to the PfSense LAN only IP? Or do I use vkernel and change the gateway from the default public IP to 10.10.10.100 (Tried this once, and I lost connectivity to ESXi server)?

David Makogon
  • 2,768
  • 1
  • 20
  • 29
Kirit Sankar Gupta
  • 41
  • 1
  • 1
  • 2
  • 2
    You need to ignore the "VMware" aspect of things here, and imagine that all your virtual kit is hardware (server, switches, uplinks, etc..). How would you do it there? – GregL Apr 12 '16 at 12:08
  • I can't set up the Internal Only LAN which will route all traffic through PfSense. That's the problem I'm facing. I don't know very well how vkernel or vswitches work. – Kirit Sankar Gupta Apr 12 '16 at 12:42
  • 1
    I have done this and when I am at my computer I will write an answer. Don't mess with the vmkernel items though. Or you can just wait for my answer – Thomas Ward Apr 12 '16 at 12:55
  • Thanks Thomas, I'll wait for your answer in that case. :) – Kirit Sankar Gupta Apr 12 '16 at 12:56
  • Is this a production system? you only have one uplink?? – Chopper3 Apr 12 '16 at 13:11
  • I added a warning to my answer, here, regarding your setup. Having only one physical NIC uplink is going to cause you headaches, because there won't be a way to force your way back into getting access to the management port in this setup. Unless you have physical machine access, and can manually redo the management network when necessary. – Thomas Ward Apr 12 '16 at 14:42
  • @Chopper3 no it's for my own private setup. Which is why I have only one uplink. – Kirit Sankar Gupta Apr 12 '16 at 16:32
  • 1
    Ok, so you've chosen to ignore our site rules that questions must be about professional environments? – Chopper3 Apr 12 '16 at 17:40
  • @Chopper3 it's not a production system. It's what our production system will be based on, on a larger scale. The reason the DC didn't give me access to two physical uplinks is because the 2nd NIC will be used "locally to be a part of VLANs". Since I'll be setting it up, I felt it would be good to try it out on a private VDS first. If I didn't I would've never known about the limitation of having a single physical uplink. I hope I haven't broken any rules. If I have, I can flag the question to a moderator. – Kirit Sankar Gupta Apr 12 '16 at 19:26

1 Answers1

16

WARNING about your setup

I would advise you caution using this system in this setup, if you have only one physical adapter and then only one uplink to the universe from the system. I would suggest going and purchasing an extra adapter for the system, and add that to your system - this way you can still have a backup physical connection.

The other headache you're going to be facing with only one uplink is that you will have to move your VMKernel to the pfSense network, and then you won't be able to have that VMKernel web-accessible. You will then need to set up VPN on the pfSense so you can VPN into the network to access the ESXi management network through the pfSense.

Some fairly important notes on my answer, of which you should take note

  • I am not going to go through how to configure pfSense. This question here asked about the ESXi network setup, so I'm addressing that part of the answer. If you need help setting up pfSense, make a separate question, or refer to one of the billion tutorials on the Internet.
  • The ESXi server I am using for my screenshots has nine (9) physical ethernet ports on it. Of those, only two are not in use, so they are on these images as vmnic5 and vmnic6. While the pictured items on my diagrams for this are not actively in use, this approach still 'works'. I've proof-of-concept tested this with both VMware Workstation (in a different but similar approach), and on my ESXi.
  • My ESXi system is ESXi 6.0U2, with the Embedded Web UI turned on. I may be switching between the WebUI and vSphere Client; in my screenshots, but the basics remain the same for configuring things.
  • My screenshots will NOT have the VMKernel on them. The VmKernel on my ESXi is on a separate vSwitch, with a separate uplink, on a separate subnet. My diagrams though will show the VMKernel item.

With my obligatory warnings out of the way, this is really not as difficult as it seems.

With physical equipment, and a physical pfSense appliance, you will have a WAN port and a LAN port. Your WAN port will be connected to the uplink to the world. The LAN port will be connected to some type of switch and the rest of your LAN infrastructure on the internal network. The pfSense WAN will be configured for the WAN IP address (static or dynamic), and the pfSense LAN will be configured for the LAN IP address with static settings, and you either turning on DHCP for the LAN or having every single system static IP'd with the default gateway pointing at that LAN.

With everything on the ESXi, we have to emulate this, but it's still the same concept.

ESXi and vSphere

Given you already have VMs, you already likely have a vSwitch0, which in turn is linked to physical system uplinks, like this, but with VMs in the "VM Network" group, and with a vmnic that actually is connected to something:

vSwitch0 - original

Importantly from this point forward, anything on vSwitch0 will now be considered on the "Uplink", as this has physical NICs on it. We can change this later. The pfSense WAN virtual NIC will be on this switch.

Create the vSwitch1, for the pfSense LAN

Here's where we'll create the LAN. If you've created vSwitches before, you can easily breeze through this part, but effectively, we are adding a vSwitch for VMs, not the VMKernel. The key here though is to not connect a physical adapter to this network, since we're running it through pfSense, which will be a virtual NIC connected into this vSwitch. I named the new port grouping here "pfSense LAN".

Create the second vSwitch

vSwitch Setup: No Physical NIC Selected!

New vSwitch in Networking config window in vSphere

Deferred until later: Move the VMs into the pfSense LAN

Because this may be a system where you cannot have downtime, I'm going to have the VMs moving to the pfSense LAN as a later step.

pfSense VM

Now, for pfSense. When you create the pfSense VM, you need to give it two virtual NICs - one for the pfSense "WAN", which is the 'outside' that the uplink exists on, and one for the pfSense "LAN", which is the internal network which the pfSense firewall protects and does NAT and routing to the uplink for the machines on the LAN.

When you set up the pfSense, you need to make sure you give the VM two virtual network adapters (vNICs, from this point forward in this answer). They each need to be on the individual network groups we have, one on vSwitch0 ("VM Network" as earlier), and one on vSwitch1 ("pfSense LAN"). I use E1000 type adapters - they seem to work well with the FreeBSD based pfSense system.

pfSense "New VM Window" for Network Settings

This is optional, but I highly recommend this. Make sure you have a VM on the "pfSense LAN" network where you can access the GUI so you can access the pfSense after you set it up. You'll need this to configure the pfSense Firewall, unless you know what you're doing on the CLI for pfSense (I will not add the pfSense Setup steps here, as that's a different question, basically.)

ONLY move to this section AFTER you've configured pfSense for the LAN and WAN settings, and set up firewall rules so you can start with an "Everything can go out to the world from the LAN" setup, and then start adding whitelist items and firewall rules before locking down.

After you've configure your pfSense, we come back to the part where you may have downtime if you're having to have access from the outside in to the VMs.

Move the VMs to the pfSense LAN

Basically, edit the network adapter settings for each of your VMs that are on the default "VM Network", and set them to be on the "pfSense LAN" network segment. If everything is static IP'd, then you should be all set, just make sure on the VMs to set the "gateway" to be the Static IP you set on the pfSense LAN adapter. (That way, the VMs can route through the pfSense).

Once you're done, and your VMs can connect to anything outside through the pfSense (by setting the gateway correctly on each machine), then you just need to configure the pfSense to permit connections inbound to the machines from the outside where necessary.

And now you're done. The pfSense should be able to reach to the 'net, and so should the VMs. If you really care about "VM Network" being present, add a group to vSwitch0 called "pfSense WAN" and then edit the pfSense network settings to have the "VM Network" vNIC connected to the new "pfSense WAN" group.

The only thing we haven't adjusted here is where the VMKernel sits. While I am not going to do this here, because it will break my system heavily, you theoretically can add a VMKernel to vSwitch1's properties, and give it a static IP in the "pfSense LAN" if you want the management to only be accessible form the "pfSense LAN" section of the network. Once you confirm that you can reach in through, there, you can remove the VMKernel on vSwitch0. If everything starts breaking, though, then you'll have to have physical access to the machine to reconfigure the management network through the machine itself.

Note that if you are already on a LAN or such where the management network resides, leave the management network alone. Unless your ESXi needs to reach out to the Internet, you don't need it to go through a gateway.

Post-configuration pfSense / ESXi Network Topology Diagram

Effectively, with regards to ESXi and the topology with regards to your VMs, pfSense, and the uplink, you will end up with something like this, focusing solely on the ESXi part.

Post-Configuration Topology with Regards to ESXi, pfSense, and VMs

Thomas Ward
  • 787
  • 2
  • 7
  • 18
  • 2
    Excellent and comprehensive answer! I appreciate the disclaimer since so many people insist on doing this with ESXi. – ewwhite Apr 12 '16 at 15:02
  • Couldn't have got a better answer, this is absolutely perfect! Thank you sir. – Kirit Sankar Gupta Apr 12 '16 at 16:32
  • @KiritSankarGupta You're welcome! I'm glad this answers your question! This is a very basic setup, though, in reality you won't have just the one uplink and just the one physical NIC, you'll have multiple NICs, and possibly two or three connections through different hardware/routes to the uplink, in order for some equipment load balancing or customized routing, or any number of other reasons. I still *strongly suggest* getting an extra NIC for your machine, there's probably a PCI express one that would fit the system that isn't too expensive. – Thomas Ward Apr 12 '16 at 17:11