Python/Pod cannot reach the internet

Question

I'm using python3 with microk8s to develop a simple web service.

The service is working properly (with docker in my local development machine), but the production machine (Ubuntu18.04 LTS with microk8s in Azure) cannot reach the internet (SMTP/Web REST API) once the pod was started (all internal service is working).

Problem

The pod cannot ping the hostname but the IP address. After investigation, the pod is working as expected except for the external resource. When executing the nslookup, it seems to be ok. But the ping is not working.

bash-5.1# ping www.google.com
ping: bad address 'www.google.com'
bash-5.1# nslookup www.google.com
Server:         10.152.183.10
Address:        10.152.183.10:53

Non-authoritative answer:
Name:   www.google.com
Address: 74.125.68.103
Name:   www.google.com
Address: 74.125.68.106
Name:   www.google.com
Address: 74.125.68.99
Name:   www.google.com
Address: 74.125.68.104
Name:   www.google.com
Address: 74.125.68.105
Name:   www.google.com
Address: 74.125.68.147

Non-authoritative answer:
Name:   www.google.com
Address: 2404:6800:4003:c02::93
Name:   www.google.com
Address: 2404:6800:4003:c02::63
Name:   www.google.com
Address: 2404:6800:4003:c02::67
Name:   www.google.com
Address: 2404:6800:4003:c02::69


bash-5.1# ping 74.125.68.103
PING 74.125.68.103 (74.125.68.103): 56 data bytes
64 bytes from 74.125.68.103: seq=0 ttl=55 time=1.448 ms
64 bytes from 74.125.68.103: seq=1 ttl=55 time=1.482 ms
^C
--- 74.125.68.103 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.448/1.465/1.482 ms

bash-5.1# python3
>>> import socket
>>> socket.gethostname()
'projects-dep-65d7b8685f-jzmxx'
>>> socket.gethostbyname('www.google.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -3] Try again

Environments/Settings

host $ #In Host
host $ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic
host $ microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dashboard
    dns
    ha-cluster
    ingress
    metrics-server
    registry
    storage
  disabled:
    ambassador
    cilium
    fluentd
    gpu
    helm
    helm3
    host-access
    istio
    jaeger
    keda
    knative
    kubeflow
    linkerd
    metallb
    multus
    portainer
    prometheus
    rbac
    traefik

# In Pod
bash-5.1 # python3
>>> import sys
>>> print({'version':sys.version, 'version-info': sys.version_info})
{'version': '3.9.3 (default, Apr  2 2021, 21:20:32) \n[GCC 10.2.1 20201203]', 'version-info': sys.version_info(major=3, minor=9, micro=3, releaselevel='final', serial=0)}
bash-5.1 # 
bash-5.1 # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local ngqy0alqbw2elndk2awonodqmd.ix.internal.cloudapp.net
nameserver 10.152.183.10
options ndots:5

Even-though the basic image "python:3-alpine", got the same result (nslookup is ok, but ping no response). — KensonMan, Jul 14 '21 at 10:30
is there a proxy server connecting your machine to the internet? — Narendra Petkar, Jul 15 '21 at 04:14

Kartik Bhiwapurkar · Answer 1 · 2021-08-04T07:44:36.013

You can confirm your pod network namespace can connect to external and internal vnet ips or not through the following commands: -

kubectl --namespace=kube-system exec -it ${KUBE-DNS-POD-NAME} -c kubedns -- sh

#run ping/or nslookup using metadata endpoint

If you restart the pod or container, it can fix the issue of hostname not resolving for external IP addresses or else, you can move the pod to a different node. Also, edit the Kubernetes dns add on master (repeat for every master) as below : -

vi /etc/kubernetes/addons/kube-dns-deployment.yaml

And change the arguments for the health container as below: -

"--cmd=nslookup bing.com 127.0.0.1 >/dev/null"
"--url=/healthz-dnsmasq"
"--cmd=nslookup bing.com 127.0.0.1:10053 >/dev/null"
"--url=/healthz-kubedns"
"--port=8080"
"--quiet"

You can also try restarting the kube coredns through the following command: -

kubectl -n kube-system rollout

This will force the kubedns container to restart if the above condition occurs.

Thanking you,

Python/Pod cannot reach the internet

Problem

Environments/Settings

1 Answers1