Kubernetes pods can't access the internet while host can

Question

I'm using a Microk8s setup with the following configuration -

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  selector:
    matchLabels:
      app: jenkins
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins
      containers:
      - name: jenkins
        image: jenkins/jenkins:2.235.1-lts-alpine
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: -Xmx2048m -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 50000
          protocol: TCP
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins
      restartPolicy: Always
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 30
      volumes:
      - name: jenkins
        persistentVolumeClaim:
          claimName: jenkins-claim

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
  resources: ["services"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
  resources: ["ingresses"]
  verbs: ["create","delete","get","list","patch","update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: jenkins

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  type: NodePort
  ports:
    - name: ui
      port: 8080
      targetPort: 8080
      protocol: TCP
    - name: slave
      port: 50000
      protocol: TCP
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: jenkins

I can access the internet from my node (host), but not from my pods. my node is an ubuntu 18.04.2 LTS machine running on vSphere, within a VPN.

in official documentation (https://microk8s.io/docs/troubleshooting) it says to either

sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent

or

sudo ufw default allow routed

both doesn't fix the problem for me.

also tried suggestions in https://github.com/ubuntu/microk8s/issues/1484 without success.

are there any network policies active that might deny egress trafic from your pods? what is the exact issue? can the images be pulled and the actual pods are having issues communicating with the outside world? do you have a CNI plugin deployed in your cluster? — meaningqo, Aug 04 '21 at 10:33
@meaningqo i uploaded the k8s configs, no policy there that should deny egress traffic. the exact issue is that my pods can't communicate with the outside world while my node can, as mentioned - so for example if i ping google.com from my node it works as expected but if i 'kubectl exec -it' into my pod and try from there it doesn't work. — Yahm Levi Firseck, Aug 04 '21 at 12:15
hmmm from what i have read on the github issue, it appears that it happens on a fresh microk8s installation as well. i thought previously that it might be an existing cluster with network policies applies already. however, this issue seems to be microk8s specific. if there is no netpol active your pods should be able to reach the internet in a standard kubernetes environment. so unfortunately i don't think i can help you further, as I have not used microk8s a lot yet — meaningqo, Aug 04 '21 at 12:23
@meaningqo ok thanks anyways. if no solution will be found the next couple of days ill switch to kubeadm. — Yahm Levi Firseck, Aug 04 '21 at 12:58
Hi @Yahm Levi Firseck. Could you check if your microk8s cluster has `dns` addon installed by running `microk8s status` ? On the pod, are you able to ping external IP (for example `ping 8.8.8.8`)? — Mikolaj S., Aug 04 '21 at 15:04
Hi @Mikolaj, I was sure I tried this already, as suggested in https://github.com/ubuntu/microk8s/issues/1484, guess not, because it solved my problem! thanks a lot! — Yahm Levi Firseck, Aug 05 '21 at 20:25
@Mikolaj --UPDATE-- turns out that ```microk8s enable dns``` by itself doesn't fix the issue, tried on a fresh VM and got same problem as before... will try to find the exact configuration where it works. — Yahm Levi Firseck, Aug 05 '21 at 20:58
@Yahm Levi Firseck, in my case, if I enabled the `dns` addon AFTER the deployment of a pod, the ping wasn't working, but if I enabled the `dns` addon BEFORE the deployment, it was working without any issue on a freshly deployed pod. — Mikolaj S., Aug 05 '21 at 21:57
@Yahm Levi Firseck, maybe it's the same case in your issue. Could you check it? — Mikolaj S., Aug 06 '21 at 07:32

score 1 · Accepted Answer · answered Aug 06 '21 at 13:15

1

In order to solve this problem on Microk8s, enable dns addon BEFORE deploying with command microk8s enable dns

answered Aug 06 '21 at 13:15

Yahm Levi Firseck

31
1
7

Even after enabling dns, the issue is still there – rhoitjadhav Oct 10 '22 at 10:20
@rhoitjadhav: If it didn't work for you, why did you accept the answer? This may lead to confusion with current visitors of the question only to learn that this is not working – hakre May 21 '23 at 10:55
@hakre Maybe there's a mistake, I have not accepted the answer at all but correct me if I'm wrong – rhoitjadhav May 22 '23 at 13:04
1

@rhoitjadhav: My mistake, sorry. I got this wrong and stand corrected. Please accept my apology for the confusion caused. – hakre May 22 '23 at 16:42

Kubernetes pods can't access the internet while host can

1 Answers1