wget : can't connect to remote host when connecting to kubernetes service in separate namespace

Question

I have setup my API for Prometheus monitoring using client instrumentation but when I check the endpoint in PromUI it is in DOWN status as below :

I checked for connectivity issues by exec into the prometheus pod and trying a wget. When I do so I get :

/prometheus $ wget metrics-clusterip.labs.svc:9216
Connecting to metrics-clusterip.labs.svc:9216 (10.XXX.XX.XXX:9216)
wget: can't connect to remote host (10.XXX.XX.XX): Connection refused

I have an existing MongoDB instance with a Prometheus exporter sidecar and its working perfectly (ie its endpoint is UP in PromUI). As a check I tried connecting to this MongoDB instance's service and it turns out the prometheus pod can indeed connect (as expected) :

wget mongodb-metrics.labs.svc:9216
Connecting to mongodb-metrics.labs.svc:9216 (10.XXX.XX.X:9216)
wget: can't open 'index.html': File exists

This is what I have also tried :

1. I tried both nslookup metrics-clusterip.svc.labs and nslookup metrics-clusterip.svc.labs:9216 from the prometheus pod but I got same error :

   nslookup metrics-clusterip.svc.labs:9216 Server: 169.XXX.XX.XX Address: 169.XXX.XX.XX:XX

   ** server can't find metrics-clusterip.svc.labs:9216: NXDOMAIN

   *** Can't find metrics-clusterip.svc.labs:9216: No answer

2. However, when I port-forward the service I can successfully query the metrics endpoint and this shows that the metrics endpoint is up in the API container :

   kubectl port-forward svc/metrics-clusterip 9216

NB:Both the API routes and the metrics endpoint are using the same port (9216)

3. Check if DNS pod is running

   kubectl get pods --namespace=kube-system -l k8s-app=kube-dns NAME READY STATUS RESTARTS AGE coredns-XXXXXXXXX-XXXXX 1/1 Running 0 294d coredns-XXXXXXXXX-XXXXX 1/1 Running 0 143d

This is the configuration for my deployment :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reg
  labels:
    app: reg
    namespace: labs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reg
      release: loki
  template:
    metadata:
      labels:
        app: reg
        release: loki
    spec:
      containers:
        - name: reg
          image: xxxxxx/sre-ops:dev-latest
          imagePullPolicy: Always
          ports:
            - name: reg
              containerPort: 9216           
          resources:
            limits:
              memory: 500Mi
            requests:
              cpu: 100m
              memory: 128Mi
      nodeSelector:
        kubernetes.io/hostname: xxxxxxxxxxxx     
      imagePullSecrets:
        - name: xxxx
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-clusterip
  namespace: labs
  labels:
    app: reg
    release: loki
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: '9216'
    prometheus.io/scrape: "true"
spec:
  type: ClusterIP
  selector:
    app: reg
    release: loki
  ports:
  - port: 9216
    targetPort: reg
    protocol: TCP
    name:  reg

And the ServiceMonitor :

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: reg
  namespace: loki
  labels:
    app: reg
    release: loki
spec:
  selector:
    matchLabels:
      app: reg
      release: loki
  endpoints:
    - port: reg
      path: /metrics
      interval: 15s
  namespaceSelector:
    matchNames:
      - "labs"

I have also tried to pipe the DNS pod logs to a file but I am not sure what I should be looking to get more detail :

kubectl logs --namespace=kube-system coredns-XXXXXX-XXXX >> logs.txt

What am I missing?