Kubernetes cluster with firewall enabled on CentOS(calico) not working

Question

I've bringing up Kubernetes cluster with calico as CNI on CentOS 7 with firewall enabled. I've master and worker nodes. I was able to bring up cluster and able to list the nodes and Kubernetes system pods, all are working fine. However I'm unable to perform dns lookup.

System configuration

Kubernetes: 1.21.1
Calico: 3.19.1
Docker: 20.10.5
CentOS 7.9
IPVS enabled
Using VXLAN based network in Calico

Firewall configuration

• The ports I've enabled are.

  - 6443/tcp
  - 2379-2381/tcp
  - 10248-10260/tcp
  - 30000-32767/tcp
  - 8285/udp
  - 8472/udp
  - 7946/udp
  - 7946/tcp
  - 7472/tcp
  - 7472/udp
  - 9100/tcp
  - 443/tcp
  # cAdvisor Port
  - 4149/tcp
  # calico
  - 179/tcp
  - 4789/udp
  - 5473/tcp
  - 9099/tcp
  - 9099/udp
  

• The enabled masquerade

• Added interfaces kube-ipvs0, vxlan.calico and docker0 to trusted zone (Though was unable to add cali* interface to trusted zone as I was unable to add regex based interface rule)

The Problem is DNS lookup doesn't work. Tried following steps to identify iptables lookup.

• Installed DNS debugger pod

    apiVersion: v1
    kind: Pod
    metadata:
      name: dnsutils
      namespace: default
    spec:
      containers:
      - name: dnsutils
        image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
        command:
          - sleep
          - "3600"
        imagePullPolicy: IfNotPresent
      restartPolicy: Always

• Enable firewall debugging on worker node on which pod got scheduled

    firewall-cmd --set-log-denied=all
    firewall-cmd --reload

• Now run nslookup

    $kubectl exec -it dnsutils -- nslookup kubernetes.default

result:

   ;; connection timed out; no servers could be reached

   command terminated with exit code 1

• Get the rule that rejected the packet

    dmesg | grep -i reject

result:

    [ 5556.708338] FINAL_REJECT: IN=calib3c61c3cba9 OUT= MAC=ee:ee:ee:ee:ee:ee:de:c8:d5:97:58:87:08:00 SRC=10.244.212.65 DST=10.96.0.10 LEN=90 TOS=0x00 PREC=0x00 TTL=64 ID=49835 PROTO=UDP SPT=52743 DPT=53 LEN=70
    [ 5561.707815] FINAL_REJECT: IN=calib3c61c3cba9 OUT= MAC=ee:ee:ee:ee:ee:ee:de:c8:d5:97:58:87:08:00 SRC=10.244.212.65 DST=10.96.0.10 LEN=90 TOS=0x00 PREC=0x00 TTL=64 ID=52640 PROTO=UDP SPT=52743 DPT=53 LEN=70
    [ 5566.708055] FINAL_REJECT: IN=calib3c61c3cba9 OUT= MAC=ee:ee:ee:ee:ee:ee:de:c8:d5:97:58:87:08:00 SRC=10.244.212.65 DST=10.96.0.10 LEN=90 TOS=0x00 PREC=0x00 TTL=64 ID=54942 PROTO=UDP SPT=52743 DPT=53 LEN=70

Answer 1

Faced a similar issue recently and it took us time to find what the actual problem is.

Problem

Even when you add ports for calico it does not work because, when you explicitly add port to firewall it blocks all other ports across interfaces. And calico creates a new network interface(vxlan.calico) which needs to be added to firewalld

Symptom

One symptom we noticed due to this was, we were able to ping pods across nodes. So icmp was working but tcp/udp did not work. Most likely you will get the same symptom.

Solution

You will need to add the network interface created by calico (check ifconfig to get exact name) to a trusted zone in firewall or create a new zone for it.

Create new zone and adding interface to it:

sudo firewall-cmd --permanent --new-zone={name}
sudo firewall-cmd --permanent --zone={name} --set-target=ACCEPT
sudo firewall-cmd --permanent --zone={name} --add-interface=vxlan.calico
sudo firewall-cmd --reload

Note - you will need to do above steps for master as well as all nodes.

Answer 2

I was able to get this working with the following steps.

sudo firewall-cmd --permanent --new-zone={name}
sudo firewall-cmd --permanent --zone={name} --set-target=ACCEPT
sudo firewall-cmd --permanent --zone={name} --add-interface=vxlan.calico

Then I looped through the calico network interfaces

for i in $(ip a | grep cali | awk -F":" '{print $2}' | awk -F"@" '{print $1}') 
do 
  sudo firewall-cmd --permanent --zone={name} --add-interface="$i"
done
sudo firewall-cmd --reload