why MASQUERADE SNAT can block localhost connection?

Question

I observed a weird behaviour on linux:

First, I clean all routes and iptables rules:

ip route flush table main
ip route flush table default
ip route flush table local
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X
ip6tables -P INPUT ACCEPT
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT
ip6tables -t nat -F
ip6tables -t mangle -F
ip6tables -F
ip6tables -X

Then I add a local route:

ip route add local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 table local

Then, on a terminal I open a port with nc -lp 12345 and on another terminal I connect to it with nc 127.0.0.1 12345 and I can send and receive data between the netcat server and client. So for now, all is good.

Now, from it and after killing the previous netcat server and client, if I run:

iptables -t nat -A POSTROUTING -j MASQUERADE

and I restart the netcat server, then the client fail to connect. Do you know why?

I notice that adding ip route add local 192.168.0.10 dev wlan0 proto kernel scope host src 192.168.0.10 table local make the netcat connection works again. However, I don't understand why the wlan0 interface (that has 192.168.0.10 as IP) can influence the loopback interface?

For information, I am using ArchLinux

Answer 1

This is an educated guess. The MASQUERADE option replaces the source IP address in IP packets with the address it decides to use. I think in this case, it replaces the source address with the interface address where default gateway is reachable.

So, if your default gateway is 192.168.0.1, the source address of the packet is replaced with 192.168.0.1. The destination is 127.0.0.1, and this doesn't work properly.

You should limit MASQUERADE to only packets where outgoing interface is the one towards default gateway.

You can do it with the following command:

iptables -t nat -A POSTROUTING -o <if> -j MASQUERADE

Answer 2

I don't know if it is a bug or an intentional fallback behavior, but from I what can see here, it has neither anything to do with the wlan0 local route you mentioned, or all the default gateway blah blah (no offense but it almost makes no sense to me) stated in the other/"correct" answer.

Normally how MASQUERADE works is, it picks the address that is configured on the outbound interface (which is determined by routing, hence POSTROUTING) for the source NAT that it performs. (If multiple addresses are assigned on the interface, it will probably pick the first one or the one that is set as the preferred source address in the corresponding route; I'm not that familiar with it and that is out-of-scope here anyway). It has nothing to do with the nexthop / gateway address of the default route. (That's not how source NAT works anyway.)

However, when it comes to the interface lo, things seems to become a bit tricky. More precisely, it does not seem to have something to do with the interface itself (apart from the fact that it will be the outbound interface because the destination is a local address), but rather the fact that, addresses in the 127.0.0.0/8 block is not of scope global. While I have no idea what is happening behind the scene, it seems that the traffic cannot "show up" if the host has no scope global IP address configured but attempt to MASQUERADE.

What I can see here is, even if you just configure an address that is valid for scope global (e.g. 192.168.0.10/32) on any interface (including lo), you'll see it works again. (The local route you mentioned will be added automatically. But I'm not seeing that adding only the route works here.)

For what it's worth, addresses and interfaces are not heavily bound together in Linux (not in a straight-forward way like, it will reply to traffics only if their destination address matches with the configured address on the inbound interface, even when IP forwarding is not of concern). So it might has something to do with that: in case MASQUERADE cannot pick a scope global address from what are configured on the outbound interface, it just pick one from any (I doubt that the precedence has something to do with default route though), if still no, it just refuse to work in some way.

In case you really need a rule that enables MASQUERADE on all interfaces but lo, you can have:

iptables -t nat -A POSTROUTING ! -o lo -j MASQUERADE

Answer 3

In addition to the other answers, I have made some other experiments. Please note that the following are only my conclusions, I didn't search inside the kernel source (but if you have documentation, please share).

Indeed, it seems that the lo interface and 127.0.0.1 follow their own rules.

For the other interfaces, here is how I believe the source IP is assigned to a packet (without speaking about MASQUERADE or things like raw sockets):

When one try to send a packet to an destimation IP, linux will search for a matching routing rule added like:

ip route add <NETWORK>/<PREFIX> dev <INTERFACE_XX> [src <SRC_IP>]

If the parameter src is provided (this seems to implies to add a rule like ip route add local <SRC_IP> dev <INTERFACE_YY> before. Note that INTERFACE_XX and INTERFACE_YY has not to be the same quite surprisingly), then the packet is sent on the interface INTERFACE_XX with the SRC_IP as the source IP.

If the parameter src is not provided, it will send packet on INTERFACE_XX and the source address is selected by search the first valid IP on an interface from the first started one to the last one (at the exeption of lo). If no IP is found, the source IP is set to 0.0.0.0. If an interface has multiple IP, it choose the first one. Quite surpisingly, this mean that a packet does not have necessarily the IP of the output interface it is send to.

I guess MASQUERADE choose the source IP in a similar way.

Please correct me is you think somethink is wrong.