linkerd Top feature only shows /healthz requests

Question

Doing Lab 7.2. Service Mesh and Ingress Controller from the Kubernetes Developer course from the Linux Foundation and there is a problem I am facing - the Top feature only shows the /healthz requests.

It is supposed to show / requests too. But does not. Would really like to troubleshoot it, but I have no idea how to even approach it.

More details

Following the course instructions I have:

1. A k8s cluster deployed on two GCE VMs
2. linkerd
3. nginx ingress controller
4. A simple LoadBalancer service off the httpd image. In effect, this is a NodePort service, since the LoadBalancer is never provisioned. The name is secondapp
5. A simple ingress object routing to the secondapp service.

I have no idea what information is useful to troubleshoot the issue. Here is some that I can think off:

Setup

Linkerd version

student@master:~$ linkerd version
Client version: stable-2.11.1
Server version: stable-2.11.1
student@master:~$

nginx ingress controller version

student@master:~$ helm list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
myingress       default         1               2022-09-28 02:09:35.031108611 +0000 UTC deployed        ingress-nginx-4.2.5     1.3.1
student@master:~$

The service list

student@master:~$ k get svc
NAME                                           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kubernetes                                     ClusterIP      10.96.0.1        <none>        443/TCP                      7d4h
myingress-ingress-nginx-controller             LoadBalancer   10.106.67.139    <pending>     80:32144/TCP,443:32610/TCP   62m
myingress-ingress-nginx-controller-admission   ClusterIP      10.107.109.117   <none>        443/TCP                      62m
nginx                                          ClusterIP      10.105.88.244    <none>        443/TCP                      3h42m
registry                                       ClusterIP      10.110.129.139   <none>        5000/TCP                     3h42m
secondapp                                      LoadBalancer   10.105.64.242    <pending>     80:32000/TCP                 111m
student@master:~$

Verifying that the ingress controller is known to linkerd

student@master:~$ k get ds myingress-ingress-nginx-controller -o json | jq .spec.template.metadata.annotations
{
  "linkerd.io/inject": "ingress"
}
student@master:~$

The secondapp pod

apiVersion: v1
kind: Pod
metadata:
  name: secondapp
  labels:
    example: second
spec:
  containers:
    - name: webserver
      image: httpd
    - name: busy
      image: busybox
      command:
        - sleep
        - "3600"

The secondapp service

student@master:~$ k get svc secondapp -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-09-28T01:21:00Z"
  name: secondapp
  namespace: default
  resourceVersion: "433221"
  uid: 9266f000-5582-4796-ba73-02375f56ce2b
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.105.64.242
  clusterIPs:
  - 10.105.64.242
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 32000
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    example: second
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}
student@master:~$

The ingress object

student@master:~$ k get ingress
NAME           CLASS    HOSTS             ADDRESS   PORTS   AGE
ingress-test   <none>   www.example.com             80      65m
student@master:~$ k get ingress ingress-test -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  creationTimestamp: "2022-09-28T02:20:03Z"
  generation: 1
  name: ingress-test
  namespace: default
  resourceVersion: "438934"
  uid: 1952a816-a3f3-42a4-b842-deb56053b168
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - backend:
          service:
            name: secondapp
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
status:
  loadBalancer: {}
student@master:~$

Testing

secondapp

student@master:~$ curl "$(curl ifconfig.io):$(k get svc secondapp '--template={{(index .spec.ports 0).nodePort}}')"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0    340      0 --:--:-- --:--:-- --:--:--   348
<html><body><h1>It works!</h1></body></html>
student@master:~$

through the ingress controller

student@master:~$ url="$(curl ifconfig.io):$(k get svc myingress-ingress-nginx-controller '--template={{(index .spec.ports 0).nodePort}}')"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15  100    15    0     0    319      0 --:--:-- --:--:-- --:--:--   319
student@master:~$ curl -H "Host: www.example.com" $url
<html><body><h1>It works!</h1></body></html>
student@master:~$

And without the Host header:

student@master:~$ curl $url
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
student@master:~$

And finally the linkerd dashboard Top snapshot:

Where are the GET / requests?

EDIT 1

So on the linkerd slack someone suggested to have a look at https://linkerd.io/2.12/tasks/using-ingress/#nginx and that made me examine my pods more carefully. It turns out one of the nginx-ingress pods could not start and it is clearly due to linkerd injection. Please, observe:

Before linkerd

student@master:~$ k get pod
NAME                                       READY   STATUS    RESTARTS      AGE
myingress-ingress-nginx-controller-gbmbg   1/1     Running   0             19m
myingress-ingress-nginx-controller-qtdhw   1/1     Running   0             3m6s
secondapp                                  2/2     Running   4 (13m ago)   12h
student@master:~$

After linkerd

student@master:~$ k get ds myingress-ingress-nginx-controller -o yaml | linkerd inject --ingress - | k apply -f -

daemonset "myingress-ingress-nginx-controller" injected

daemonset.apps/myingress-ingress-nginx-controller configured
student@master:~$

And checking the pods:

student@master:~$ k get pod
NAME                                       READY   STATUS    RESTARTS        AGE
myingress-ingress-nginx-controller-gbmbg   1/1     Running   0               40m
myingress-ingress-nginx-controller-xhj5m   1/2     Running   8 (5m59s ago)   17m
secondapp                                  2/2     Running   4 (34m ago)     12h
student@master:~$

student@master:~$ k describe pod myingress-ingress-nginx-controller-xhj5m |tail
  Normal   Created            19m                   kubelet            Created container linkerd-proxy
  Normal   Started            19m                   kubelet            Started container linkerd-proxy
  Normal   Pulled             18m (x2 over 19m)     kubelet            Container image "registry.k8s.io/ingress-nginx/controller:v1.3.1@sha256:54f7fe2c6c5a9db9a0ebf1131797109bb7a4d91f56b9b362bde2abd237dd1974" already present on machine
  Normal   Created            18m (x2 over 19m)     kubelet            Created container controller
  Normal   Started            18m (x2 over 19m)     kubelet            Started container controller
  Warning  FailedPreStopHook  18m                   kubelet            Exec lifecycle hook ([/wait-shutdown]) for Container "controller" in Pod "myingress-ingress-nginx-controller-xhj5m_default(93dd0189-091f-4c56-a197-33991932d66d)" failed - error: command '/wait-shutdown' exited with 137: , message: ""
  Warning  Unhealthy          18m (x6 over 19m)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 502
  Normal   Killing            18m                   kubelet            Container controller failed liveness probe, will be restarted
  Warning  Unhealthy          14m (x30 over 19m)    kubelet            Liveness probe failed: HTTP probe failed with statuscode: 502
  Warning  BackOff            4m29s (x41 over 14m)  kubelet            Back-off restarting failed container
student@master:~$

I will process the link I was given on the linkerd slack and update this post with any new findings.

Answer 1

The solution was provided by the Axenow user on the linkerd2 slack forum. The problem is that ingress-nginx cannot share the namespace with the services it provides the ingress functionality to. In my case all of them were in the default namespace.

To quote Axenow:

When you deploy nginx, by default it send traffic to the pod directly. To fix it you have to make this configuration: https://linkerd.io/2.12/tasks/using-ingress/#nginx

To elaborate, one has to update the values.yaml file of the downloaded ingress-nginx helm chart to make sure the following is true:

controller:
  replicaCount: 2
  service:
    externalTrafficPolicy: Cluster
  podAnnotations:
    linkerd.io/inject: enabled

And install the controller in a dedicated namespace:

helm upgrade --install --create-namespace --namespace ingress-nginx -f values.yaml ingress-nginx  ingress-nginx/ingress-nginx

(Having uninstalled the previous installation, of course)