iptable's ip forwarding results in weird behaviour

Question

My target is simple : Build a high volume, fast, low latency soft switch using IP forwarding. I have a server test 'Server' and 2 test clients : 'Client1' & 'Client2'. 'Client1:c1' is sending udp packets to 'Server:s1' & 'Client2:c2' is sending udp packets to 'Server:s2'. I just wanna forward 'Client:c1' packets to 'Client:c2' and vice versa. Wouldn't mind if this is possible using only 1 port in sevrer. 'Client1' & 'Client2' is currently a simple udp sender thread which is sending udp packet continiously. Now I am applying rule :

sudo iptables --table nat --append PREROUTING --protocol udp --destination 'Server' --dport 's1' --jump DNAT --to-destination 'Client2:c2'

sudo iptables --table nat --append PREROUTING --protocol udp --destination 'Server' --dport 's2' --jump DNAT --to-destination 'Client1:c1'

sudo iptables --table nat --append POSTROUTING --protocol udp --destination 'Client2' --dport 'c2' --jump SNAT --to-source 'Server:s0'

sudo iptables --table nat --append POSTROUTING --protocol udp --destination 'Client1' --dport 'c1' --jump SNAT --to-source 'Server:s0' 

And If I apply this rule at once and then start both 'Client1' & 'Client2' thread, it works just fine. Even if I delete all rules at once it also works fine but the problem starts if I start adding/deleting this rules 1 by 1 in the time where 'Client1' & 'Client2' is continiously sending data.

behaviour 1 : 'Client1' & 'Client2' running (nothing is happening as no rules are applied) and then I apply the 4 rules to 'Server' at once, then it should work just like previous, but not.. now 'Server' is only forwarding 'Client1' packets to 'Client2' and even SNAT isn't working for that. though after some indefinite time it works though. Other behaviour is just like this, when I am applying some rule to 'Server' iptable, the rule isn't taking effect immediately if 'Client1' & 'Client2' is running.

behaviour 2 : It also happened to me that I flushed the nat table and server was still forwarding for some time.

behaviour 3 : when applying/deleting per rule 1 by 1 while 'Client1' & 'Client2' is still sending/receiving, it wasn't taking effect randomly

I searched and found out in several post that iptables rule takes effect immediately, but for our case it isn't, specially some rules working and some not suddenly and what we are doing wrong here? is there some kind of cache 'Server' iptable maintaing or does it need reload though I found there is no option for reloading? ** I am not interested in 'socat' as it takes the packets to user-space which has bottleneck for high volume packets.** Thanks in advance for any help..!

Answer 1

The following will redirect anything going from client1:port1 to server:ports to be delivered to the client2:port2, where it appears as coming from the server:ports:

iptables -t nat -A PREROUTING -p udp -s client1 --sport port1 -d server --dport ports -j DNAT --to-destination client2:port2
iptables -t nat -A POSTROUTING -p udp -s client1 --sport port1 -d client2 --dport port2 -j SNAT --to-source server:ports

The first rule changes the destination from server:ports to client2:port2 (because it is DNAT rule), the second rule changes the source from client1:port1 to server:ports. By the time packet is going through second rule, it has the destination already changed, but source isn't, so the matches in the rule might look strange. Replies will undergo the reverse translation automatically.

If you want conversations initiated on client2 (not replies) to work similarly, you need to add the mirror rules:

iptables -t nat -A PREROUTING -p udp -s client2 --sport port2 -d server --dport ports -j DNAT --to-destination client1:port1
iptables -t nat -A POSTROUTING -p udp -s client2 --sport port2 -d client1 --dport port1 -j SNAT --to-source server:ports

If server has any filtering, you'll need to add rules permitting forwarded packets to pass:

iptables -t filter -A FORWARD -p udp -s client1 --sport port1 -d client2 --dport port2 -j ACCEPT
iptables -t filter -A FORWARD -p udp -s client2 --sport port2 -d client1 --dport port1 -j ACCEPT

Addresses appear half-translated here too, because the packets always traverse the firewall in the following order: PREROUTING, FORWARD, POSTROUTING.

Now the client1 will be in the full confidence that it talks with server:ports (at least, on the network level; of course packets could contain something that hints on who is really answering). The same is true for client2, who'll be sure that it talks with server:ports.

Also remember, that once the conversation had begun, the firewall remembers its state; the removal or addition of rules won't have any immediate effect on established connections, because the nat table is always traversed only by the first packet in the connection; the result of the translation is stored in the state table which is used to translate replies and subsequent packets. That state table could be seen by cat /proc/net/nf_conntrack or using utilities from the conntrack-tools package. In case of UDP, the "connection" is defined by the addresses, ports and timeout in the conversation, and "ends" when no packet is seen in any direction for a certain amount of time, 30 seconds by default.

I also want to state this in clear: the forwarding or routing is never performed by the iptables. The routing and forwarding will be performed regardless of any iptables rules in place, according to the routing policy database rules and routing tables contents. It is simply incorrect to say "iptables's ip forwarding", as iptables only sets translations. It configures the in-kernel firewall filtering rules and network address and port translation rules, e.g. how the kernel will change addresses in packets, which apparently could influence its routing decisions (because routing decisions are based on addresses).