0

I am trying to expose a service in a simple kubernetes cluster composed of a single worker and one master. In particular, I am using the descriptor below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
spec:
  selector:
    app: mongodb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017

Then I try to use this service from another pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo-express
  labels:
    app: mongo-express
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo-express
  template:
    metadata:
      labels:
        app: mongo-express
    spec:
      containers:
      - name: mongo-express
        image: mongo-express
        ports:
        - containerPort: 8081
        env:
        - name: ME_CONFIG_MONGODB_SERVER
          value: mongodb-service

However, what I get in the other mongo-express pod is that mongodb-service cannot be resolved. In fact if I spin up a pod and I try a simple wget this is the output:

$ wget http://mongodb-service/ -O-
--2021-06-23 13:31:08--  http://mongodb-service/
Resolving mongodb-service (mongodb-service)... failed: Name or service not known.
wget: unable to resolve host address 'mongodb-service'

Instead nslookup mongodb-service works fine:

$ nslookup mongodb-service
Server:     10.96.0.10
Address:    10.96.0.10#53

However, if I try with netcat I get this:

$nc mongodb-service 27017
nc: getaddrinfo for host "mongodb-service" port 27017: Name or service not known

So it seems that it is getaddrinfo that is failing.

How could I debug the problem?

Roberto
  • 1,281
  • 3
  • 16
  • 23
  • Apparently your cluster DNS is not working properly. When I run it on my **GKE** cluster, everything works just fine. Could you share more info about your particular k8s cluster ? How was it set up ? Were you able to connect to other pods via a service or is it your very first test on this cluster ? You can set up a simple nginx deployment, expose it with ClusterIP Service and try to connect to it from a different pod. Same result as with mongo ? What about the result of running `kubectl get pods -n kube-system | grep dns` ? – mario Jun 24 '21 at 12:12
  • The cluster was created following this guide: https://www.linuxtechi.com/install-kubernetes-k8s-on-ubuntu-20-04/. Running the command you suggest shows two coredns pods both running and with one container each. – Roberto Jun 24 '21 at 12:45
  • Hi @Roberto, any progress with this issue ? Do you experience the same with other workloads on this k8s cluster or only with mongodb ? – mario Aug 12 '21 at 21:25

1 Answers1

0

Go inside the pod and try following:

nslookup mongodb-service.default.svc.cluster.local

It will give you the clusterIP of service and then try to wget with this IP instead of service's DNS.

wget http://10.100.142.166 -O-

The reason for using IP over DNS: https://stackoverflow.com/questions/53921281/kubernetes-1-13-coredns-cluster-curl-service

Alif Biswas
  • 348
  • 1
  • 7
  • I have added a few more info. Using the hardcoded IP address is not really an option here. – Roberto Jun 23 '21 at 15:19
  • In case this could be useful, the kubectl get service command gives me 10.109.178.205 as IP address for that service which is different from the ns lookup output. Using 10.109.178.205 in the yaml solves my problem, but there is clearly something wrong somewhere. – Roberto Jun 23 '21 at 16:52
  • @Roberto, but note that `nslookup` in your example doesn't return the IP of your `mongodb-service` at all. It shows only the IP address of the DNS server that is being queried, which in your case happens to be `10.96.0.10` (note `#53` which is DNS port). The correct output may look something like this: `bash-5.0# nslookup mongodb-service Server: 10.16.0.10 Address: 10.16.0.10:53 Name: mongodb-service.default.svc.cluster.local Address: 10.16.11.185` – mario Jun 24 '21 at 12:17