0

I'm facing this issue for the last 3 days and still can't figure out why I can't access NodePort outside k8s cluster using NodePort service type.

Basically I have set up a 3 node cluster in rhel8, I followed this tutorial https://www.tecmint.com/install-a-kubernetes-cluster-on-centos-8/ but faced some issues in weavenet network that's why I'm doing it again but this time with calico.

Here's my 3 node cluster: enter image description here

I installed calico latest version for the network addon and as you can see all kube pods are healthy and running.

enter image description here

Now here's my deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
    name: hello-k8s
spec:
    selector:
     matchLabels:
        app: hello-k8s
    replicas: 1
    template:
      metadata:
        labels:
          app: hello-k8s
      spec:
        containers:
         - name: spring-boot
           image: fuzzy28/hello-k8s:v3
           ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
    name: hello-k8s-svc
spec:
    selector:
      app: hello-k8s
    ports:
      - protocol: TCP
        port: 8080
        targetPort: 8080
    type: NodePort

I deployed it and it's running fine in the node that is highlighted below.

enter image description here

If we check the service, it's running in 32020 port.

I opened the ports in all nodes using below command

firewall-cmd --permanent --add-port=30000-32767/tcp

enter image description here

So the moment of truth, after checking this on the browser outside the k8s cluster it's not accessible. I tried this in all nodes IP address but not accessible. enter image description here

The weird thing is the URL is accessible only inside the pod from where it's deployed.

enter image description here

I really don't understand what's happening here, here's my network interfaces in case you want to see.

enter image description here

Ruelos Joel
  • 103
  • 4

1 Answers1

1

It took me a while to find out whats the most probable cause of the issue. It turns out that since the RHEL 8 release iptables are deprecated and nftables are the new replacement for it.

38.2. When to use firewalld, nftables, or iptables

  • iptables: The iptables utility is deprecated in Red Hat Enterprise Linux 8. Use instead nftables.

Kubernetes networking uses iptables and it's not compatible with nftables.

I tried to solve your issue by disabling firewalld and nftables, and force rhel8 to use iptables instead but it still did not solved all the connectivity issues in the cluster.

Seems like the best you can do at the moment is to use older rhel version. Let me know if you have any further questions.

acid_fuji
  • 573
  • 3
  • 8
  • oh you were able to replicate the issue? that's what I was thinking in the very beginning. if should i use rhel8 or not. thanks a lot man! I found similar issue here https://medium.com/@liau.weijie/how-to-install-docker-and-kubernetes-cluster-on-redhat-8-centos-8-f774fc071e82 – Ruelos Joel Jul 08 '20 at 14:45
  • I downgraded my OS to rhel7 and all kube-pods are healthy and running but now nodeport service is only accessible on the node from where the pod is deployed. If I do worker-node-2-ip:port, service is not reachable. Only worker-node-1-ip:port from where pod is deployed works. is it normal? I'm using weavenet this time – Ruelos Joel Jul 09 '20 at 05:05
  • @RuelosJoel, to answer your first comment: Yes, I was able to replicate and I encountered the same issue (I created couple of clusters for that). For the second comment: This is not expected behavior so I think you will to have to place another question with steps to reproduce etc. If you haven`t tried that, you may want to check and disable [firwealld and enable iptables](https://www.thegeekdiary.com/how-to-disable-firewalld-and-and-switch-to-iptables-in-centos-rhel-7/). – acid_fuji Jul 09 '20 at 07:52
  • 1
    as a workaround what I did, i just iptables forward accept to all nodes and now it's working fine! thanks man! – Ruelos Joel Jul 09 '20 at 10:36