4

I'm trying to create a multisite network using pfsense linked together with Tinc VPN. This is my current topology:

    Router A                Router B
****************        ****************
*              *        *              *
* 10.0.0.1/16  *--------* 10.1.0.1/16  *
*              *        *              *
****************        ****************
       |                        |
       |                        |
****************        ****************
*              *        *              *
* 10.0.0.11/16 *        * 10.1.0.16/16 *
*              *        *              *
****************        ****************
    Node 1                 My Desktop

My desktop and both routers can hit every machine on the network, but node 1 can only hit router A.

Router B is currently sitting behind Verizon's router with the VPN port forwarded. The link is established. Although, I doubt this would be the root of the issue here.

I've triple checked my pfsense configs and they are identical to one another. I'm really not sure what is preventing node 1 from communicating with the rest of the network. I've basically opened everything up. I have any to any rules for all interfaces yet node 1 cannot find the route.

If it is of any relevance, router A and node 1 are hosted in the cloud through Vultr. I have the private network enabled and node 1 is requesting addresses from the DHCP server on router A. Vultr does assign private IPs in the 10.X.X.X space with the same subnet. Could my IP space be conflicting with theirs? Vultr does not deploy a gateway and the IPs they assign are entirely static.

You can use any IPs you like on the private network. We assign one IP by default, but you can ignore it and use other ones if you like.

I really don't know why node 1 cannot hit the other subnet and I'm hoping that someone might be able to help me figure that out.

10.1.0.16

bkvaluemeal@Formula:~$ ping -c 3 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=0.330 ms
64 bytes from 10.1.0.1: icmp_seq=2 ttl=64 time=0.319 ms
64 bytes from 10.1.0.1: icmp_seq=3 ttl=64 time=0.305 ms

--- 10.1.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.305/0.318/0.330/0.010 ms
bkvaluemeal@Formula:~$ ping -c 3 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=63 time=9.82 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=63 time=8.86 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=63 time=38.0 ms

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 8.864/18.925/38.085/13.553 ms
bkvaluemeal@Formula:~$ ping -c 3 10.0.0.11
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_seq=1 ttl=62 time=11.5 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=62 time=10.5 ms
64 bytes from 10.0.0.11: icmp_seq=3 ttl=62 time=9.37 ms

--- 10.0.0.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 9.370/10.482/11.555/0.892 ms

bkvaluemeal@Formula:~$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether e0:3f:49:ad:81:03 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.16/16 brd 10.1.255.255 scope global dynamic eno1
       valid_lft 6915sec preferred_lft 6915sec
    inet6 fe80::20dc:2028:faee:5420/64 scope link 
       valid_lft forever preferred_lft forever
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 54:27:1e:55:ae:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.163/24 brd 192.168.1.255 scope global dynamic wlp3s0
       valid_lft 76214sec preferred_lft 76214sec
    inet6 fe80::d9de:6606:5307:968b/64 scope link 
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 52:54:00:d1:33:dd brd ff:ff:ff:ff:ff:ff
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:82:c6:99:06 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

10.1.0.1

PING 10.1.0.16 (10.1.0.16): 56 data bytes
64 bytes from 10.1.0.16: icmp_seq=0 ttl=64 time=0.177 ms
64 bytes from 10.1.0.16: icmp_seq=1 ttl=64 time=0.312 ms
64 bytes from 10.1.0.16: icmp_seq=2 ttl=64 time=0.194 ms

--- 10.1.0.16 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.177/0.228/0.312/0.060 ms

PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=8.926 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=8.335 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=8.290 ms

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.290/8.517/8.926/0.290 ms

PING 10.0.0.11 (10.0.0.11): 56 data bytes
64 bytes from 10.0.0.11: icmp_seq=0 ttl=63 time=11.052 ms
64 bytes from 10.0.0.11: icmp_seq=1 ttl=63 time=9.573 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=63 time=9.913 ms

--- 10.0.0.11 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.573/10.179/11.052/0.632 ms

10.0.0.1

PING 10.1.0.16 (10.1.0.16): 56 data bytes
64 bytes from 10.1.0.16: icmp_seq=0 ttl=63 time=8.307 ms
64 bytes from 10.1.0.16: icmp_seq=1 ttl=63 time=9.256 ms
64 bytes from 10.1.0.16: icmp_seq=2 ttl=63 time=9.109 ms

--- 10.1.0.16 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.307/8.891/9.256/0.417 ms

PING 10.1.0.1 (10.1.0.1): 56 data bytes
64 bytes from 10.1.0.1: icmp_seq=0 ttl=64 time=8.618 ms
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=8.579 ms
64 bytes from 10.1.0.1: icmp_seq=2 ttl=64 time=8.702 ms

--- 10.1.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.579/8.633/8.702/0.051 ms

PING 10.0.0.11 (10.0.0.11): 56 data bytes
64 bytes from 10.0.0.11: icmp_seq=0 ttl=64 time=1.142 ms
64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=2.385 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=2.053 ms

--- 10.0.0.11 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.142/1.860/2.385/0.525 ms

10.0.0.11

root@node1:~# ping -c 3 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1 icmp_seq=1 ttl=64 time=1.10 ms
64 bytes from 10.0.0.1 icmp_seq=2 ttl=64 time=1.04 ms
64 bytes from 10.0.0.1 icmp_seq=3 ttl=64 time=0.749 ms

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.749/0.968/1.106/0.156 ms

root@node1:~# ping -c 3 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.

--- 10.1.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2042ms

root@node1:~# ping -c 3 10.1.0.16
PING 10.1.0.16 (10.1.0.16) 56(84) bytes of data.

--- 10.1.0.16 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2048ms

root@node1:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether XX:XX:XX:XX:XX:XX brd XX:XX:XX:XX:XX:XX
    inet 45.77.X.X/23 brd 45.77.X.X scope global ens3
       valid_lft forever preferred_lft forever
    inet6 2001:19f0:X:X:X:X:X:X/64 scope global mngtmpaddr dynamic 
       valid_lft 2591544sec preferred_lft 604344sec
    inet6 fe80::5400:X:X:X/64 scope link 
       valid_lft forever preferred_lft forever
3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 5a:01:01:3c:13:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.11/16 brd 10.0.255.255 scope global ens7
       valid_lft forever preferred_lft forever
    inet6 fe80::5801:1ff:fe3c:13c8/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:65:df:2f:a1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:65ff:fedf:2fa1/64 scope link 
       valid_lft forever preferred_lft forever

Router A NAT Mapping

Router B NAT Mapping

Firewall floating rules

Firewall pkg_tinc rules

Firewall WAN rules

Firewall LAN rules

Router A IPv4 Routes

Router B IPv4 Routes

bkvaluemeal
  • 43
  • 6
  • Can you be a little more specific about how many devices are on each subnet. I’m confused by your post. Are we talking about a single device on the subnet that can’t reach the other subnet, or many devices on that network that can’t reach the other? Are you saying you can ping node 1 from the desktop but not vice versa? – Appleoddity Oct 25 '17 at 00:39
  • Also, it might help to visualize thing by posting the ping command and output from each machine. Also an IP config on each and a route list. – Appleoddity Oct 25 '17 at 00:40
  • @Appleoddity This is literally it. There are only 4 devices on my test network. Now, connected to the Verizon router there are several devices, but they are on a completely different subnet and outside of this scope. – bkvaluemeal Oct 25 '17 at 00:41
  • @Appleoddity You are correct that it is just the one host (node 1). I'd deploy another host to test on Vultr, but my trial only allows me to spin up a max of 2 hosts. – bkvaluemeal Oct 25 '17 at 00:58
  • As the ping go from one side, Desktop --> Node1 but not when launched from Node1 --> Desktop, I suspect Router A drop the packet, or Router B. (interface thrust or rule in in example) – yagmoth555 Oct 25 '17 at 01:05
  • Which ones doing NAT... :/ – Jacob Evans Oct 25 '17 at 01:10
  • Both routers are NATting to the other subnets. Hold on. I'll upload an image of the mappings. – bkvaluemeal Oct 25 '17 at 01:12
  • If the networks are not the same, you really don't want to NAT. NAT is something you only want to do if you need to. – Ron Maupin Oct 25 '17 at 01:17
  • @RonMaupin I should note that Tinc is configured in router mode. So, if NAT is not preferred, what is the preferred way to link these two networks? My knowledge of NAT is limited. Look at the images I uploaded. I'm not entirely sure if I am actually NATting. – bkvaluemeal Oct 25 '17 at 01:19
  • @yagmoth555 How can I validate dropped packets? – bkvaluemeal Oct 25 '17 at 01:20
  • Routers route between networks. NAT is not required for routing. NAT is simply translating either the source, destination, or both addresses. Routers are simply a convenient place to NAT, but routing happens without NAT all the time. – Ron Maupin Oct 25 '17 at 01:38
  • @RonMaupin Adding a gateway and a static route is preventing me from hitting the pfsense portal. – bkvaluemeal Oct 25 '17 at 02:06
  • There are routes between the 2 routers. See added images. – bkvaluemeal Oct 25 '17 at 02:16
  • Realizing now that I forgot to sanitize router A's public IP. Whatever. I'll be rebuilding these hosts after the issue is resolved. – bkvaluemeal Oct 25 '17 at 02:25

1 Answers1

5

"the ping go from one side, Desktop --> Node1 but not when launched from Node1 --> Desktop" - This statement, if accurate, eliminates all possibilities of a routing issue.

In order for a ping to work from Desktop->Node1, you also have to receive a reply back from Node1->Desktop. That indicates everything is setup fine on the VPN and routing.

Instead this is a firewall issue. Because the ping from Node1 is dropped at Router B, yet Router B can ping Desktop, then the firewall issue is probably on Router B. Router B is allowing outbound and "related" connections but is not allowing inbound connections.

Based on the information you posted I would have to say it has to do with what you have called "Firewall LAN rules." Change it to ANY/ANY/ANY/ANY (or whatever) for testing. The firewall rules are ambiguous so I can't say one way or another.

EDIT:

We discovered that Node 1 had two network interfaces. One on the "private network" with IP 10.0.0.11 and another with a public IP address AND default gateway. In addition, the NAT rules were causing the traffic to be NATted across the VPN tunnel. Therefore, Desktop could successfully PING Node 1 because the traffic arriving at Node 1 appeared to be coming from 10.0.0.1. But when trying to PING Desktop from Node1, Node1 did not have a route to 10.1.0.0/16.

Once we cleaned up the routing table on Node1 and set the NAT rules to automatic on Router A and B, everything started working as intended.

Appleoddity
  • 3,488
  • 2
  • 13
  • 33
  • Wait, but, I already have any everything rules for all. I moved them up in the list, but still no dice. – bkvaluemeal Oct 25 '17 at 02:23
  • Yes, I understand, but I don't recognize the configuration interface you took screenshots of, so I don't know exactly what I am looking at. The NAT rules look funky too. But, because they look "symmetrical" on both sides, I'm assuming they are OK because it works in one direction. They don't look "perfect" but it has to be functional enough to get a ping from one side to another and a reply back. So, this has to be a firewall issue on Router B. If it's not that, then there is some other piece to this puzzle we are not seeing. Perhaps whatever is between Router A and Router B. – Appleoddity Oct 25 '17 at 02:31
  • [This](https://imgur.com/a/3sjE8) is what pfsense's rules configuration looks like. – bkvaluemeal Oct 25 '17 at 02:38
  • I also just DMZed router B through Verizon's portal. I'm not ready to deploy this thing just yet. – bkvaluemeal Oct 25 '17 at 02:42
  • Join me in chat... https://chat.stackexchange.com/rooms/info/67623/bkvaluemeal-vpn-issue?tab=general (not sure if I did it right. Chat looks new) – Appleoddity Oct 25 '17 at 02:51