Kubelet still sending requests even after springboot readiness probes is in REFUSING_TRAFFIC stage

Question

I was trying readiness probe in kubernetes with a springboot app. After the app starts, lets say after 60 seconds I fire ReadinessState.REFUSING_TRAFFIC app event.

I use port-forward for kubernetes service(Cluster-Ip) and checked /actuator/health/readiness and see "status":"OUT_OF_SERVICE" after 60 seconds.

I, then fire some GET/POST requests to service.

Expected: Service unavailable message

Actual: GET/POST endpoints return data as usual

Is the behavior expected. Please comment.

Sample liveness/readiness probe yaml

        livenessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Authorization
              value: Basic xxxxxxxxxxxxxx
            path: /actuator/health/liveness
            port: http
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 10
        name: sample-app
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Authorization
              value: Basic xxxxxxxxxxxxxx
            path: /actuator/health/readiness
            port: http
            scheme: HTTP
          initialDelaySeconds: 140
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 10

Answer 1

This is expected behavior as:

• $ kubectl port-forward service/SERVICE_NAME LOCAL_PORT:TARGET_PORT

is not considering the state of the Pod when doing a port forwarding (directly connects to a Pod).

---

Explanation

There is already a great answer which pointed me on further investigation here:

• Stackoverflow.com: Answer: Does kubectl port-forward ignore loadBalance services?

Let's assume that you have a Deployment with a readinessProbe (in this example probe will never succeed):

• $ kubectl get pods, svc (redacted not needed part)

NAME                                     READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-64ff4d8749-7khht    0/1     Running   0          97m
pod/nginx-deployment-64ff4d8749-bklnf    0/1     Running   0          97m
pod/nginx-deployment-64ff4d8749-gsmml    0/1     Running   0          97m

NAME                         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/nginx                ClusterIP   10.32.31.105   <none>        80/TCP      97m

• $ kubectl describe endpoints nginx

Name:         nginx
Namespace:    default
Labels:       <none>
Annotations:  <none>
Subsets:
  Addresses:          <none>
  NotReadyAddresses:  10.36.0.62,10.36.0.63,10.36.0.64 # <-- IMPORTANT
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  80    TCP

Events:  <none>

As you can see all of the Pods are not in Ready state and the Service will not send the traffic to it. This can be seen in a following scenario ( create a test Pod that will try to curl the Service):

• $ kubectl run -it --rm nginx-check --image=nginx -- /bin/bash
• $ curl nginx.default.svc.cluster.local

curl: (7) Failed to connect to nginx.default.svc.cluster.local port 80: Connection refused

Using kubectl port-forward:

• $ kubectl port-forward service/nginx 8080:80
• $ curl localhost:8080

<-- REDACTED --> 
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<-- REDACTED -->

More light can be shed on why it happened by getting more verbose output from the command:

• kubectl port-forward service/nginx 8080:80 -v=6 (the number can be higher)

I0606 21:29:24.986382    7556 loader.go:375] Config loaded from file:  /SOME_PATH/.kube/config
I0606 21:29:25.041784    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/services/nginx 200 OK in 51 milliseconds
I0606 21:29:25.061334    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/pods?labelSelector=app%3Dnginx 200 OK in 18 milliseconds
I0606 21:29:25.098363    7556 round_trippers.go:444] GET https://API_IP/api/v1/namespaces/default/pods/nginx-deployment-64ff4d8749-7khht 200 OK in 18 milliseconds
I0606 21:29:25.164402    7556 round_trippers.go:444] POST https://API_IP/api/v1/namespaces/default/pods/nginx-deployment-64ff4d8749-7khht/portforward 101 Switching Protocols in 62 milliseconds
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

What happened:

• kubectl requested the information about the Service: nginx
• kubectl used the selector associated with the Service and looked for Pods with the same selector (nginx)
• kubectl chose a single Pod and port-forwarded to it.

The Nginx welcome page showed as the port-forward connected directly to a Pod and not to a Service.

---

Additional reference:

• Github.com: Kubernetes: Issues: kubectl port-forward should allow forwarding to a Service

• $ kubectl port-forward --help

# Listen on port 8443 locally, forwarding to the targetPort of the service's port named "https" in a pod selected by the service

kubectl port-forward service/myservice 8443:https