I have a setup where I have multiple interface aliases (with private IPs) for the primary network interface (eth0).
eth0 Link encap:Ethernet HWaddr 0a:ed:01:17:53:b4
inet addr:10.0.1.220 Bcast:10.0.1.255 Mask:255.255.255.0
eth0:0 Link encap:Ethernet HWaddr 0a:ed:01:17:53:b4
inet addr:192.168.1.1 Bcast:0.0.0.0 Mask:255.255.255.0
eth0:1 Link encap:Ethernet HWaddr 0a:ed:01:17:53:b4
inet addr:192.168.1.2 Bcast:0.0.0.0 Mask:255.255.255.0
I am applying different network shaping policies to each of these aliases separately. In order for processes bound to each of these private IPs to be able to talk to the internet, I am using iptables as a NAT to masquerade the IP address to that of eth0
iptables -t nat -A POSTROUTING -s 192.168.1.1/24 -o eth0 -j MASQUERADE
This works great. If I watch the traffic leaving eth0, I can see that the source IP is properly changed.
# Send pings using eth0:0
ping -I 192.168.1.1 www.google.com
# Monitor packets sent out of eth0
tcpdump -i eth0 icmp
# 21:34:18.664664 IP ip-10-0-1-220.ec2.internal > ord37s08-in-f4.1e100.net: ICMP echo request, id 3957, seq 2, length 64
# 21:34:18.683022 IP ord37s08-in-f4.1e100.net > ip-10-0-1-220.ec2.internal: ICMP echo reply, id 3957, seq 2, length 64
# 21:34:19.666137 IP ip-10-0-1-220.ec2.internal > ord37s08-in-f4.1e100.net: ICMP echo request, id 3957, seq 3, length 64
# 21:34:19.683954 IP ord37s08-in-f4.1e100.net > ip-10-0-1-220.ec2.internal: ICMP echo reply, id 3957, seq 3, length 64
# 21:34:28.774873 IP ip-10-0-1-220.ec2.internal > ord37s08-in-f4.1e100.net: ICMP echo request, id 3962, seq 1, length 64
# 21:34:28.793410 IP ord37s08-in-f4.1e100.net > ip-10-0-1-220.ec2.internal: ICMP echo reply, id 3962, seq 1, length 64
For traffic shaping, I'm using dummynet+ipfw. When I go to create the necessary traffic shaping policies for each of my aliases.
# Apply an outbound bandwidth cap of 1Mbps to eth0:0
ipfw pipe 1 config bw 1Mbit/s
ipfw queue 1 config pipe 1 queue 100
ipfw add queue 1 ip from 192.168.1.1 to any
Then the packets generated by dummynet are never actually seen by iptables, and therefore when the traffic leaves eth0, it has the internal private IP which obviously is never going to get a reply.
# Send pings using eth0:0
ping -I 192.168.1.1 www.google.com
# Monitor packets sent out of eth0
tcpdump -i eth0 icmp
# 21:34:29.776431 IP 192.168.1.1 > ord37s08-in-f4.1e100.net: ICMP echo request, id 3962, seq 2, length 64
# 21:34:30.784435 IP 192.168.1.1 > ord37s08-in-f4.1e100.net: ICMP echo request, id 3962, seq 3, length 64
# 21:34:31.792440 IP 192.168.1.1 > ord37s08-in-f4.1e100.net: ICMP echo request, id 3962, seq 4, length 64
# 21:34:32.800446 IP 192.168.1.1 > ord37s08-in-f4.1e100.net: ICMP echo request, id 3962, seq 5, length 64
When I look at iptables -t nat -vnL I can confirm that the packets are never seen by iptables.
I realize that both ipfw and iptables are using the underlying netfilter library to handle packets, and based on my reading, they both hook into the POSTROUTING hooks. Is it possible that they are competing in some way? If so, can I somehow "re-order" the rules in netfilter such that dummynet will reinject the packet back into the flow so that it's picked up by iptables? Is there an alternative way to create a NAT for these private IPs so that I can avoid this conflict?
Unfortunately, the NAT functionality of ipfw seems to have been dropped when porting it to Linux, so that's not a possibility.
Update
When digging into the dummynet code, it seems as though it re-injects the delayed packets into the stack using dst_output (here) which based on my reading should call ip_output which in turn should execute the NF_INET_POST_ROUTING netfilter hook.
When adding some TRACE iptables rules,
iptables -A PREROUTING -t raw -p icmp -j TRACE
iptables -A OUTPUT -t raw -p icmp -j TRACE
I got the following.
Without dummynet running (but using the interface alias):
[23416.053128] TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37956 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=1 UID=0 GID=0
[23416.053136] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37956 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=1 UID=0 GID=0
[23416.053139] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37956 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=1 UID=0 GID=0
[23416.053143] TRACE: nat:POSTROUTING:rule:1 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=37956 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=1 UID=0 GID=0
[23416.055612] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=0a:0d:f8:70:35:b4:0a:86:45:e6:5c:49:08:00 SRC=172.217.7.164 DST=10.0.1.255 LEN=84 TOS=0x00 PREC=0x00 TTL=47 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=12659 SEQ=1
[23416.055628] TRACE: filter:INPUT:policy:1 IN=eth0 OUT= MAC=0a:0d:f8:70:35:b4:0a:86:45:e6:5c:49:08:00 SRC=172.217.7.164 DST=192.168.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=47 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=12659 SEQ=1
[23417.054547] TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=38092 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=2 UID=0 GID=0
[23417.054557] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=38092 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=2 UID=0 GID=0
[23417.055950] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=0a:0d:f8:70:35:b4:0a:86:45:e6:5c:49:08:00 SRC=172.217.7.164 DST=10.0.1.255 LEN=84 TOS=0x00 PREC=0x00 TTL=47 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=12659 SEQ=2
[23417.055969] TRACE: filter:INPUT:policy:1 IN=eth0 OUT= MAC=0a:0d:f8:70:35:b4:0a:86:45:e6:5c:49:08:00 SRC=172.217.7.164 DST=192.168.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=47 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=12659 SEQ=2
[23418.056141] TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=38099 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=3 UID=0 GID=0
[23418.056157] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=172.217.7.164 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=38099 DF PROTO=ICMP TYPE=8 CODE=0 ID=12659 SEQ=3 UID=0 GID=0
[23418.057635] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=0a:0d:f8:70:35:b4:0a:86:45:e6:5c:49:08:00 SRC=172.217.7.164 DST=10.0.1.255 LEN=84 TOS=0x00 PREC=0x00 TTL=47 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=12659 SEQ=3
[23418.057646] TRACE: filter:INPUT:policy:1 IN=eth0 OUT= MAC=0a:0d:f8:70:35:b4:0a:86:45:e6:5c:49:08:00 SRC=172.217.7.164 DST=192.168.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=47 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=12659 SEQ=3
And with dummynet running (you can see that the POSTROUTING NAT policy is never applied.
[23535.355649] TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43670 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=1 UID=0 GID=0
[23535.355657] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43670 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=1 UID=0 GID=0
[23535.355661] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43670 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=1 UID=0 GID=0
[23536.362312] TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43689 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=2 UID=0 GID=0
[23536.362324] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43689 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=2 UID=0 GID=0
[23536.362330] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43689 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=2 UID=0 GID=0
[23537.370345] TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43819 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=3 UID=0 GID=0
[23537.370379] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43819 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=3 UID=0 GID=0
[23537.370385] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=192.168.1.1 DST=216.58.217.132 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43819 DF PROTO=ICMP TYPE=8 CODE=0 ID=12715 SEQ=3 UID=0 GID=0
