1

I have a problem I cant figure out, I am trying to access webserver on [RPI], but packets never reach iptables FILTER chain

I will try to explain it a bit:
GW1 has public address and doing DNAT from port 8080 to 192.168.69.14:80
S1 is a openVPN server eth0 for LAN and tap0 for VPN are bridged
[RPI] has running webserver on port 80
GW2 is default gateway for [RPI] and has no public address

     GW1 <-----------------> S1 <-----Open VPN tunnel------> [RPI] <--Default route--> GW2
(192.168.69.1)        (192.168.69.22)                 (192.168.69.14 - tap0)       (192.168.30.1)
                                                      (192.168.30.2 - wlan0)

Now, everything works fine If I reach [RPI]'s webserver from S1, GW2 or do a ping from GW1
But if I try to access webserver via GW1's public IP on port 8080, packets do reach [RPI], but disappear in iptables., as you can see here, also iptables rules are printed below:

Apr 27 18:13:51 WeatherStorm kernel: [11383.698445] TRACE: raw:PREROUTING:rule:3 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=275 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215BF0000000001030307)
Apr 27 18:13:51 WeatherStorm kernel: [11383.874415] TRACE: raw:PREROUTING:policy:4 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=275 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215BF0000000001030307)
Apr 27 18:13:51 WeatherStorm kernel: [11384.051167] TRACE: mangle:PREROUTING:policy:1 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=275 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215BF0000000001030307)
Apr 27 18:13:51 WeatherStorm kernel: [11384.227423] TRACE: nat:PREROUTING:policy:1 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=275 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215BF0000000001030307)
Apr 27 18:13:52 WeatherStorm kernel: [11384.459821] TRACE: raw:PREROUTING:rule:3 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=22576 DF PROTO=TCP SPT=33137 DPT=80 SEQ=671043022 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215FE0000000001030307)
Apr 27 18:13:52 WeatherStorm kernel: [11384.635037] TRACE: raw:PREROUTING:policy:4 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=22576 DF PROTO=TCP SPT=33137 DPT=80 SEQ=671043022 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215FE0000000001030307)
Apr 27 18:13:52 WeatherStorm kernel: [11384.811610] TRACE: mangle:PREROUTING:policy:1 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=22576 DF PROTO=TCP SPT=33137 DPT=80 SEQ=671043022 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215FE0000000001030307)
Apr 27 18:13:52 WeatherStorm kernel: [11384.988901] TRACE: nat:PREROUTING:policy:1 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=22576 DF PROTO=TCP SPT=33137 DPT=80 SEQ=671043022 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B215FE0000000001030307)
Apr 27 18:13:54 WeatherStorm kernel: [11386.698855] TRACE: raw:PREROUTING:rule:3 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=276 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B218AD0000000001030307)
Apr 27 18:13:54 WeatherStorm kernel: [11386.874488] TRACE: raw:PREROUTING:policy:4 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=276 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B218AD0000000001030307)
Apr 27 18:13:54 WeatherStorm kernel: [11387.050505] TRACE: mangle:PREROUTING:policy:1 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=276 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B218AD0000000001030307)
Apr 27 18:13:54 WeatherStorm kernel: [11387.228835] TRACE: nat:PREROUTING:policy:1 IN=tap0 OUT= MAC=82:ed:f9:e6:c2:ea:00:0c:42:f5:XX:XX:XX:XX SRC=37.188.XXX.XXX DST=192.168.69.14 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=276 DF PROTO=TCP SPT=33136 DPT=80 SEQ=3502183596 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405360402080A01B218AD0000000001030307)

Raw:

[root@WeatherStorm tmp]# iptables -L -nv -t raw
Chain PREROUTING (policy ACCEPT 5750 packets, 748K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   27  1620 TRACE      tcp  --  *      *       37.188.XXX.XXX       0.0.0.0/0           
  270 15120 TRACE      icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
   51  3958 TRACE      tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80

    Chain OUTPUT (policy ACCEPT 4768 packets, 911K bytes)
     pkts bytes target     prot opt in     out     source               destination         
        8   448 TRACE      icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
        0     0 TRACE      tcp  --  *      *       0.0.0.0/0            37.188.XXX.XXX

Mangle:

[root@WeatherStorm tmp]# iptables -L -nv -t mangle
Chain PREROUTING (policy ACCEPT 4177 packets, 544K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 3661 packets, 374K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 3498 packets, 674K bytes)
 pkts bytes target     prot opt in     out     source               destination  

Chain POSTROUTING (policy ACCEPT 3498 packets, 674K bytes)
 pkts bytes target     prot opt in     out     source               destination

NAT:

[root@WeatherStorm tmp]# iptables -L -nv -t nat
Chain PREROUTING (policy ACCEPT 596 packets, 180K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 80 packets, 9600 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 59 packets, 4443 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 59 packets, 4443 bytes)
 pkts bytes target     prot opt in     out     source               destination

Filter: 

[root@WeatherStorm tmp]# iptables -L -nv -t filter
    Chain INPUT (policy ACCEPT 23788 packets, 2365K bytes)
     pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 23777 packets, 5142K bytes)
 pkts bytes target     prot opt in     out     source               destination

Any idea what/where could be a problem? If I tried access from GW2 it looks like

Apr 27 18:22:02 WeatherStorm kernel: [11873.756818] TRACE: raw:PREROUTING:policy:4 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34259 DF PROTO=TCP SPT=38739 DPT=80 SEQ=3482087833 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F44A0000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11873.850894] TRACE: mangle:PREROUTING:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34259 DF PROTO=TCP SPT=38739 DPT=80 SEQ=3482087833 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F44A0000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11873.945646] TRACE: nat:PREROUTING:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34259 DF PROTO=TCP SPT=38739 DPT=80 SEQ=3482087833 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F44A0000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11874.039622] TRACE: mangle:INPUT:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34259 DF PROTO=TCP SPT=38739 DPT=80 SEQ=3482087833 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F44A0000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11874.133002] TRACE: filter:INPUT:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34259 DF PROTO=TCP SPT=38739 DPT=80 SEQ=3482087833 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F44A0000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11874.226404] TRACE: nat:INPUT:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34259 DF PROTO=TCP SPT=38739 DPT=80 SEQ=3482087833 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F44A0000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11874.319744] TRACE: raw:PREROUTING:policy:4 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22857 DF PROTO=TCP SPT=38740 DPT=80 SEQ=3524767168 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F4890000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11874.413794] TRACE: mangle:PREROUTING:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22857 DF PROTO=TCP SPT=38740 DPT=80 SEQ=3524767168 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F4890000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11874.508565] TRACE: nat:PREROUTING:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22857 DF PROTO=TCP SPT=38740 DPT=80 SEQ=3524767168 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F4890000000001030307)
Apr 27 18:22:02 WeatherStorm kernel: [11874.602511] TRACE: mangle:INPUT:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22857 DF PROTO=TCP SPT=38740 DPT=80 SEQ=3524767168 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F4890000000001030307)
Apr 27 18:22:04 WeatherStorm kernel: [11874.695929] TRACE: filter:INPUT:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22857 DF PROTO=TCP SPT=38740 DPT=80 SEQ=3524767168 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F4890000000001030307)
Apr 27 18:22:04 WeatherStorm kernel: [11874.789331] TRACE: nat:INPUT:policy:1 IN=wlan0 OUT= MAC=a0:f3:c1:2f:86:6a:00:1c:bf:8e:XX:XX:XX:XX SRC=192.168.30.1 DST=192.168.30.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22857 DF PROTO=TCP SPT=38740 DPT=80 SEQ=3524767168 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A01B3F4890000000001030307)

.

Solution:


Solved with following iptables rules on S1

iptables -A PREROUTING -t mangle -i br0 -p tcp --dport 80 -d 192.168.69.14 -j MARK --set-mark 1
iptables -A POSTROUTING -t nat --match mark --mark 1 -j SNAT --to-source 192.168.69.1
Ency
  • 1,231
  • 1
  • 19
  • 27

1 Answers1

2

Your routing configuration for [RPI] is incorrect — if you want it to be reachable from public Internet addresses via GW1, either the default route must point to GW1, or (if outgoing connections from [RPI] to public Internet addresses must go through GW2) you need to configure policy routing on [RPI], so that packets which belong to incoming connections through GW1 are routed via GW1.

(The third option is to make GW1 also SNAT the packets to 192.168.69.1, so that they will come to [RPI] from a directly connected IP, but in this case you will not be able to determine the actual client address on [RPI] when handling those packets. For HTTP you can work around this by installing a reverse HTTP proxy on GW1 instead of using NAT, and passing the real client IP in HTTP headers.)

In your current configuration packets coming from tap0 which do not come from directly connected hosts may be dropped due to enabled rp_filter on that interface (although, according to ip-sysctl.txt, the default value of rp_filter is 0, lots of distributions enable it by default in their networking configuration). If you look at the netfilter packet flow diagram, rp_filter is applied at the “routing decision” node, which is consistent with your observations (the last node you see is nat:PREROUTING, which is just before the “routing decision” node).

Note that just disabling rp_filter will not help you, because reply packets from [RPI] will then be just sent via GW2, and even if they will somehow reach the other host, it will reject them, because those packets will not have proper NAT processing done on them by GW1. You really must make sure that replies to NATed packets which came through GW1 are sent back to GW1 — without this NAT will not work properly.

Sergey Vlasov
  • 6,288
  • 1
  • 21
  • 30