I have created a kubernetes service that uses an NLB with the following annotations
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn...
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
I am able to see that the NLB is able to terminate TLS, it presents the correct certificate and completes the handshake with the client. However, I get an empty response from my backend. When I don't terminate TLS at the NLB everything is fine, I get a valid response, and my backend presents the certificate instead.
My suspicion is that communication is breaking down between the NLB and the backend pods serving the traffic. What certificate is being used for TLS between the backend pods and the NLB in Kubernetes?