Assigned second public IP to VM from outside not reachable

Question

I have a dedicated server running proxmox with a mixed configuration, NAT based network configuration for all VM's and routed network (brouter according to hetzner docs) (see below) for one particular VM. One NIC one public main IP (94.x.x.A) and an extra ordered second public IP (94.x.x.B) basically on the same interface. I also have a virtualised opnsense/pfsense VM which uses vmbr10 and vmbr11 as WAN and LAN.

auto enp0s31f6
iface enp0s31f6 inet static
  address 94.x.x.A
  netmask 255.255.255.192
  pointopoint 94.x.x.x
  gateway 94.x.x.x
  mtu 1500
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -d 94.x.x.A -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -d 94.x.x.A -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2

auto vmbr1
iface vmbr1 inet static
   address 94.x.x.A
   netmask 255.255.255.255
   bridge_ports none
   bridge_stp off
   bridge_fd 0
   pre-up brctl addbr vmbr1
   up ip route add 94.x.x.B/32 dev vmbr1
   down ip route del 94.x.x.B/32 dev vmbr1
# BBB
iface vmbr1 inet6 static
   address 2xxx:x:x:x::10
   netmask 64
# BBB IPv6


auto vmbr10
iface vmbr10 inet static
        address 10.10.10.1/29
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j SNAT --to-source 94.x.x.A
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.2/32' -o enp0s31f6 -j SNAT --to-source 94.x.x.A
        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING  -i fwbr+ -j CT --zone 1
# OPNSense WAN - Proxmox LAN

iface vmbr10 inet6 static
        address 2xx:x:x:x::3/125
        up ip route add ...


auto vmbr11
iface vmbr11 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
# VM Net

Now I created another linux bridge (vmbr1) and moved this one VM with Big Blue Button out of vmbr11, not going through opnsense/pfsense anymore and I would really like to forward everything to it. Iptables should absolutely not care about anything, blindly forward everything arriving at that second public IP 94.x.x.B to my local BBB VM. So like the VM itself is directly connected to the internet. It basically owns that IP. With the current setup everything seems to work but only when I'm connected to my VPN (a vpn inside vmbr11 on that host), so the connection to vmbr1 and the BBB VM works, I can start a session, the microphone/ web cam works. Also the BBB checks are working fine:

docker exec -it bbb-docker-greenlight-1 bundle exec rake conf:check

Checking environment: Passed
Checking Connection: Passed
Checking Secret: Passed
Checking SMTP connection: Passed

as well as curl --trace-ascii - -k https://94.x.x.B:443/bigbluebutton/api shows a success reply.

So until here, I would say, the docker deployment and setup/configuration of BBB was successful. Now when I try to access the url from the internet (not connected to vpn) the browser can't open the page, simply not reachable. I disabled the proxmox firewall, but does not help. I also can't ping bbb.my-domain.com or the IP from outside, although the ping command resolves to the correct IP. I guess I'm reaching the correct server, but then there is a small thing missing on the /etc/network/interfaces file.

Does anyone see what is missing to make it available from the internet?

Answer 1

I'm not sure how exactly your current setup works so I'll just write down my own design and see if it helps.

First of all, given that you have a sense VM that seems to be supposed to serve as the (default) gateway for most your VMs, on the host side (proxmox) the physical NIC should be enslaved to (a.k.a. become a bridge port of) a "WAN bridge", with the sense VM connected to both the "WAN bridge" and a "LAN bridge", which the other VMs are connected to.

With such setup, all the IP configurations should be done inside the VMs (including the WAN/public IP and LAN/private IP of the sense VM). No IP configuration should be done on any of the interfaces on the host side. The physical NIC and the two bridges should merely be "up", period. (You can assign a LAN/private IP on the LAN bridge if you want to allow communication between the host and the VMs though, but that IP should NOT be used as the default gateway on the VMs, otherwise the sense VM will be pointless.)

Certainly the problem with such setup is that the host itself will have no Internet connectivity, since it has no public IP, unless:

1. You have an extra public IP which you can assigned on the "WAN bridge", OR,
2. Add a default route that uses the "LAN IP" of the sense VM as the gateway

Now since you do have an extra public IP and your "new" goal is to use it as some kind of "DMZ" for one particular VM , so unless there's some particular reason that you don't want any traffics from the host itself to the Internet to have that extra public IP (but instead, the other one assigned to the sense VM) as their source IP, option 1 would be the way to go.

Note that whether you need a new bridge (and in that case, a new private IP subnet) purely depends on the whether or not you want that one VM to be isolated (well, on L2) from the other VMs. If not, it can remain attached to the same bridge. (Just make sure the LAN bridge has a private IP and that VM uses that IP instead of the LAN IP of the sense VM as its default gateway.)

P.S. If you need the proxmox host itself to use the public IP assigned to the sense VM for Internet, while at the same time you need the "DMZ" setup for "that one VM", you probably need to set up source-based routing with ip rule, which can allow traffics from that VM to look up an alternate route table with default route that uses the "WAN bridge" and the corresponding "public gateway" of your public IPs, while traffics originated from the host itself can look up the main table with default route that uses the sense VM as the default gateway like the other VMs do as mentioned above.

---

(Certainly SNAT should be set up on Proxmox for the special VM, i.e. private IP Z or S, as well so that "active" traffics from the VM would "work".)

Answer 2

I finally made it, therefore I'm summing everything up in here, which helped me to make Docker based BigBlueButton work on a Proxmox VM. I lost a lot of time to get it work, no need that it happens to you as well. I tried first with running behind NAT, using opnsense/pfsene (1:1 NAT + reflection etc.), tried everything and wasn't successful. Either the checks failed or the Microphone/ webcam wasn't working. Now everything works.

The following Hetzner docs were crucial, I used the routed conf for my Proxmox Host and the Rocky Linux VM:

• https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve/de?title=Proxmox_VE#netzwerkkonfiguration-hostsystem-routed
• Debian: https://docs.hetzner.com/robot/dedicated-server/network/net-config-debian-ubuntu#routed-brouter
• CentOS: https://docs.hetzner.com/robot/dedicated-server/network/net-config-cent-os/#routed-brouter

I encountered some differences between them (CentOS based doc is missing GATEWAY entry in the Guest section, Debian doc contains pre-up brctl addbr virbr1 in the host section). You definitely need the GATEWAY entry, I cannot say if brctl addbr ... is doing something important, haven't added that one. I would say, sticking to the community doc according to the information in there, is your best bet.

1. Order a second public IP from your provider and setup the Host and guest like in the docs (Be careful, watch closely to not make a mistake for subnet, gateway, pointtopoint etc. entries). This will actually assign the second public IP to your VM. That means, I'm still using NAT etc. for my main IP. This new IP is not going through the NAT, it is bound to a new linux bridge and leads directly to your VM. IMPORTANT: Be careful that you don't have any post-up iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to x.x.x.x which would intercept your traffic and DNAT it to somewhere else.
2. Configure the VM's firewall

firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --permanent --zone=public --add-service=https
firewall-cmd --permanent --zone=public --add-port=16384-32768/udp
firewall-cmd --reload

3. deploy and configure BBB like described in https://github.com/bigbluebutton/docker:

# used the following settings
greenlight: y
https proxy: n
coturn: n
...

Side note: logs can be found inside the greenlight container: /usr/src/app/log/production.log

4. For using my own SSL certificate, I added an extra nginx container which I can easily configure as a reverse proxy to use my ssl certificates and proxy to BBB via proxy_pass http://127.0.0.1:48087;. I couldn't find anything official in the docs how to re-configure the already present nginx to use my own certificates. I didn't want to patch anything into it, which could break easily or cause any issues when upgrading BBB. It's not that pretty, I know, but you can try to re-configure the present nginx/greenlight whatever is needed and let us know if it works.

5. TODO: Still missing fail2ban for the login

That's it, use the following checks:

curl --trace-ascii - -k https://94.x.x.B:443/bigbluebutton/api
docker exec -it bbb-docker-greenlight-1 bundle exec rake conf:check

which should both be successful. Your BigBlueButton should now fully work.