the PREROUTING rules set by the calico is anywhere

Question

I'm confused. This is my iptables nat table config

[root@k8s-51 woniu.zhang]# iptables -t nat -L -v  --line-numbers

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    4566K  396M cali-PREROUTING  all  --  any    any     anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
2    4567K  396M KUBE-SERVICES  all  --  any    any     anywhere             anywhere             /* kubernetes service portals */
3     7687  465K CNI-HOSTPORT-DNAT  all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
4     3923  236K DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
5     142K   12M            all  --  any    any     anywhere             anywhere
6     142K   12M            all  --  any    any     anywhere             anywhere

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    7901K  549M cali-OUTPUT  all  --  any    any     anywhere             anywhere             /* cali:tVnHkvAo15HuiPy0 */
2    7902K  549M KUBE-SERVICES  all  --  any    any     anywhere             anywhere             /* kubernetes service portals */
3     555K   33M CNI-HOSTPORT-DNAT  all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
4       67  4237 DOCKER     all  --  any    any     anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    6657K  469M cali-POSTROUTING  all  --  any    any     anywhere             anywhere             /* cali:O3lYWMrLQYEMJtB5 */
2        0     0 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere
3    7256K  507M CNI-HOSTPORT-MASQ  all  --  any    any     anywhere             anywhere             /* CNI portfwd requiring masquerade */
4    8073K  560M KUBE-POSTROUTING  all  --  any    any     anywhere             anywhere             /* kubernetes postrouting rules */

Chain CNI-HOSTPORT-DNAT (2 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain CNI-HOSTPORT-MASQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       11   660 MASQUERADE  all  --  any    any     anywhere             anywhere             mark match 0x2000/0x2000

Chain CNI-HOSTPORT-SETMARK (0 references)
num   pkts bytes target     prot opt in     out     source               destination
1       11   660 MARK       all  --  any    any     anywhere             anywhere             /* CNI portfwd masquerade mark */ MARK or 0x2000

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  docker0 any     anywhere             anywhere

Chain KUBE-FIREWALL (0 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-DROP  all  --  any    any     anywhere             anywhere

Chain KUBE-KUBELET-CANARY (0 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain KUBE-LOAD-BALANCER (0 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  any    any     anywhere             anywhere

Chain KUBE-MARK-DROP (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain KUBE-MARK-MASQ (3 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MARK       all  --  any    any     anywhere             anywhere             MARK or 0x4000

Chain KUBE-NODE-PORT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  any    any     anywhere             anywhere             /* Kubernetes nodeport TCP port for masquerade purpose */ match-set KUBE-NODE-PORT-TCP dst

Chain KUBE-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MASQUERADE  all  --  any    any     anywhere             anywhere             /* Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose */ match-set KUBE-LOOP-BACK dst,dst,src
2        0     0 RETURN     all  --  any    any     anywhere             anywhere             mark match ! 0x4000/0x4000
3        0     0 MARK       all  --  any    any     anywhere             anywhere             MARK xor 0x4000
4        0     0 MASQUERADE  all  --  any    any     anywhere             anywhere             /* kubernetes service traffic requiring SNAT */

Chain KUBE-SERVICES (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  any    any     anywhere             anywhere             /* Kubernetes service cluster ip + port for masquerade purpose */ match-set KUBE-CLUSTER-IP dst,dst
2        0     0 KUBE-NODE-PORT  all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
3        0     0 ACCEPT     all  --  any    any     anywhere             anywhere             match-set KUBE-CLUSTER-IP dst,dst

Chain cali-OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1    7901K  549M cali-fip-dnat  all  --  any    any     anywhere             anywhere             /* cali:GBTAv2p5CwevEyJm */

Chain cali-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1    7933K  551M cali-fip-snat  all  --  any    any     anywhere             anywhere             /* cali:Z-c7XtVd2Bq7s_hA */
2    7933K  551M cali-nat-outgoing  all  --  any    any     anywhere             anywhere             /* cali:nYKhEzDlr11Jccal */
3        0     0 MASQUERADE  all  --  any    tunl0   anywhere             anywhere             /* cali:JHlpT-eSqR1TvyYm */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL

Chain cali-PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1    4566K  396M cali-fip-dnat  all  --  any    any     anywhere             anywhere             /* cali:r6XmIziWUJsdOK6Z */

Chain cali-fip-dnat (2 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-fip-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-nat-outgoing (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     2185  131K MASQUERADE  all  --  any    any     anywhere             anywhere             /* cali:Dw4T8UWPnCLxRJiI */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst

The iptables-save results as below

[root@k8s-51 woniu.zhang]# iptables-save

# Completed on Tue Jan 12 11:11:06 2021
# Generated by iptables-save v1.4.21 on Tue Jan 12 11:11:06 2021
*nat
:PREROUTING ACCEPT [4:463]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [25:1810]
:POSTROUTING ACCEPT [25:1810]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:DOCKER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
:cali-OUTPUT - [0:0]
:cali-POSTROUTING - [0:0]
:cali-PREROUTING - [0:0]
:cali-fip-dnat - [0:0]
:cali-fip-snat - [0:0]
:cali-nat-outgoing - [0:0]
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING
-A PREROUTING
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "cali:O3lYWMrLQYEMJtB5" -j cali-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A DOCKER -i docker0 -j RETURN
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE
-A KUBE-SERVICES -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:GBTAv2p5CwevEyJm" -j cali-fip-dnat
-A cali-POSTROUTING -m comment --comment "cali:Z-c7XtVd2Bq7s_hA" -j cali-fip-snat
-A cali-POSTROUTING -m comment --comment "cali:nYKhEzDlr11Jccal" -j cali-nat-outgoing
-A cali-POSTROUTING -o tunl0 -m comment --comment "cali:JHlpT-eSqR1TvyYm" -m addrtype ! --src-type LOCAL --limit-iface-out -m addrtype --src-type LOCAL -j MASQUERADE
-A cali-PREROUTING -m comment --comment "cali:r6XmIziWUJsdOK6Z" -j cali-fip-dnat
-A cali-nat-outgoing -m comment --comment "cali:Dw4T8UWPnCLxRJiI" -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -j MASQUERADE
COMMIT

I'm confused the previous two anywhere rules:

[root@k8s-51 woniu.zhang]# iptables -t nat -L -v  --line-numbers

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    4566K  396M cali-PREROUTING  all  --  any    any     anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
2    4567K  396M KUBE-SERVICES  all  --  any    any     anywhere             anywhere             /* kubernetes service portals */

The first rule has accept all traffic, how and when the following rules match?

Could you please show output of `iptables-save` instead? That might be somewhat harder to read, but it is the complete unprocessed configuration that is in the kernel, more definitive source. — Nikita Kipriyanov, Jan 11 '21 at 21:11

score 0 · Answer 1 · answered Jan 12 '21 at 06:24

No, the first rule doesn't accepts all traffic. It just directs packet into another chain. More, if no rules matched, or the packed is accepted, that only finished packet travel through this table and this master chain, but it still must go through other chains of the table and other tables.

For this case: it seems only nat table is in use, where incoming packet travels rules in the following order:

it enters PREROUTING chain,
after the rule -A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING it jumps into cali-PREROUTING
there is a rule -A cali-PREROUTING -m comment --comment "cali:r6XmIziWUJsdOK6Z" -j cali-fip-dnat it jumps to cali-fip-dnat
there is no rules in that chain, so it returns eventually to the chain PREROUTING and processes next rule
the rule -A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES puts it into KUBE-SERVICES

There some useful processing begins. Packet either is being marked, or is accepted (and if this is complete firewall, no further processing is done).

and so on.

Also note, this traversing is done for only first packet of a "connection" (bidirectional stream of related packets). Whet Linux determines the fate for this packet, it becomes a fate of this "connection". It installs a dynamic record into a special conntrack table, and if some following packet matches this connection by conntrack, it gets processed according to the dynamic record in the conntrack, and not being fully processed through the firewall rules. The dynamic record in the conntrack gets deleted after connection end, either by close (liker TCP FIN or RST) or after timeout.

I basically clear, Thank u so much – woniu Jan 17 '21 at 02:40 — woniu, Jan 17 '21 at 02:40

the PREROUTING rules set by the calico is anywhere

1 Answers1