6

Is it possible to directly expose a server behind a L4 load balancer, with a public certificate?

This server is inside a Kubernetes pod. There is a TCP loadbalancer service in front of it which creates the external L4 LB.

My problem is that the TLS traffic does not reach the container inside the pod. So if you succeeded with a similar configuration, I would be interested into knowing.

Update

I did not mention that the traffic is GRPC.

Here is what I did: I have a domain and a corresponding official certificate. I want to secure the grpc connection.

I tried two approches:

  • with google ESP container, I put the cert as an nginx secret, pass it to the container, set an ssl-port. Behind the ESP in the same pod, I have my grpc server

In this case I get a message like this on the client side:

D0610 14:38:46.246248584 32401 security_handshaker.cc:176] Security handshake failed: {"created":"@1591792726.246234613","description":"Handshake failed","file":"../deps/grpc/src/core/lib/security/transport/security_handshaker.cc","file_line":291,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}

I see some TLS exchanges with wireshark but no log in esp.

  • without ESP, I put the cert in my GRPC server. There the GRPC server fails with something like this:

error:1408F10B:SSL routines:ssl3_get_record:wrong version number

In the google ESP documentation I see that I have to prove the domain belongs to me and upload the cert (but where)?

Update 2

As of today, I see no evidence that it is feasible.

IMO, the main issue is that the L4 has the IP corresponding to the domain name of the certificate. Hence the pods don't have the correct IP to prove that they can use the cert so their request towards roots are denied (I have no proof of that as I can't get debug info from nginx in the ESP. I have seen a request with the pure GRPC server solution though).

halfer
  • 19,824
  • 17
  • 99
  • 186
unludo
  • 4,912
  • 7
  • 47
  • 71
  • It definitely is possible. What have you done so far? – suren Jun 09 '20 at 19:28
  • The L4 will not terminate the TLS connection, it will forward it to your pod. Your service needs to expose port 443 and your pod must allow for SSL termination – Patrick W Jun 10 '20 at 02:21
  • And your pod need to host the TLS certificate (with Apache or nginx for example) for performing the communication handshake – guillaume blaquiere Jun 10 '20 at 07:40
  • @suren thanks for your feedback, I updated my question – unludo Jun 10 '20 at 14:14
  • So, just to confirm, you want to expose a gRPC server, that is inside a pod, with TLS termination at pod level, exposed as LoadBalancer in GKE cluster, is it correct?Also, share the link where you saw that ESP requires you to prove the cert is yours by uploading it. All links that you followed are good as well. I'm trying to reproduce your issue and this info would be very valuable, Otherwise I'll provide you an example for gRPC + ESP on GKE ok? – Will R.O.F. Jun 10 '20 at 16:47
  • @willrof Yes I try both solutions (with or without ESP). I put the link regarding the proof you own the domain in the question. I'll be interested in your example for sure! – unludo Jun 10 '20 at 17:15
  • Hey, I'm trying to reproduce your issue...Specially using a domain cert not self signed is a tricky one. But one thing I can help you as of now: - https://cloud.google.com/endpoints/docs/grpc/verify-domain-name here is how you verify your domain name! – Will R.O.F. Jun 16 '20 at 16:42
  • @willrof Hi, thanks for trying. Yes regarding the verify domain, I already did it. The thing is now the https call to the endpoint works (it just returns some text indicating there is no service there but it shows the cert is working). The remaining issue is in GRPC with the cert, the handshake won't happen. I suspect an issue with the cert or even the root_cert as my cert comes from Gandi. – unludo Jun 17 '20 at 08:45
  • Other tip from SSL on ESP: You must specify both `CN` and `subjectAltName` in your server certificate. It should match the DNS or IP used by the clients to call your service. Otherwise SSL handshake will fail. Since you suspect about an issue with the cert, It would be very valuable if you could test with other cert, even a self signed (I found out this method to create one with CN and AltSubName very straight forward: https://security.stackexchange.com/a/159537) – Will R.O.F. Jun 17 '20 at 10:39

1 Answers1

2

The issue was in TLS exchange.

By installing the cert in the ESP, it works fine with a web browser and the certificate is indicated valid, whereas with a GRPC client, the TLS handshake fails. Adding additional trace info helped.

By checking my certificate (not self signed but attached to my domain), I found that there is an intermediate certificate provided with it. I installed it along with the domain certificate (in the same crt file) and then it worked.

I don't know exactly why it is behaving like this but maybe it's due to the root_cert file in grpc client lib being too old.

By the way for a domain cert, there is no specific requirement regarding CN and subjectAltName for the certificate. It works without it. So it must only apply to self signed certs as I have seen elsewhere.

I had another issue that disturbed my task: be careful not to name the service port of the L4 load balancer with 'http2' inside. I had some side effect, making another deployment fail due to this. In fact when you do https, don't put http2 in the name.

Anyway it is now working and answers the request for the bounty. Thanks to all who tried to help :)

unludo
  • 4,912
  • 7
  • 47
  • 71
  • Normally, the server has to provide the full certificate chain, but some TLS clients (e.g. anything based off of Microsoft's SecureChannel) still accept the cert without its chain if they can find the intermediate & root cert in their certificate store because this is such a common deployment error. In other words, your browser was lying when it said the cert was OK. Other TLS clients (e.g. anything based off of OpenSSL) are more strict and will reject it. An easy way to check this is to run `openssl s_client -showcerts -connect $host:$port` and look for validation errors in the output. – André Caron Nov 18 '20 at 14:01