1

I am trying to setup an nginx kubernetes ingress. I am able to serve http and websockets content on different routes at the moment.

However I am not able to add GRPC routes on the same host. Adding this annotation nginx.ingress.kubernetes.io/backend-protocol: "GRPC" breaks the existing routes.

My java GRPC client exits with Caused by: io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f According to https://github.com/grpc/grpc-java/issues/2905 this means the request is seen as HTTP

Is there a way to have http/websocket/grpc routes on the same host using the nginx kubernetes ingress? Alternatively, is there another ingress with which this would work?

nha
  • 17,623
  • 13
  • 87
  • 133
  • Did you try add the GRPC routes on another Ingress for the same host? It is allowed to have different routes to the same host in multiple Ingress. The Ingress controller will serve multiple Ingress without problems, usually, even at the same host. – AndD Apr 28 '21 at 06:03
  • Could you provide [a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example) ? – Matt Apr 28 '21 at 08:15
  • @AndD I have not, my understanding is that this would incur additional costs (creation of a load balancer on public clouds). It may be worth a try still, thanks. – nha Apr 28 '21 at 12:23
  • @Matt taking this official example which works with http paths ( https://github.com/kubernetes/ingress-nginx/blob/ada2300c7b76a46ce37093a699b4920f706b9982/docs/examples/tls-termination/ingress.yaml ) and adding this annotation https://github.com/kubernetes/ingress-nginx/blob/ada2300c7b76a46ce37093a699b4920f706b9982/docs/examples/grpc/ingress.yaml#L7 is enough to break the ingress – nha Apr 28 '21 at 12:25
  • 1
    @nha From my experience on bare metal clusters, if you specify multiple Ingress resources with the same Ingress Class, the same Ingress Controller serves them. What basically happens is that the Nginx Ingress Controller compose the nginx.conf with all the Ingress resources defined on the cluster. You can see on the docs a list of how it behaves in building the model ( https://kubernetes.github.io/ingress-nginx/how-it-works/#building-the-nginx-model ) – AndD Apr 28 '21 at 12:47
  • After testing with a small example that seems to be the case indeed! Thanks @AndD this should be enough to get me going :) If you want to write this as an answer I will gladly accept it – nha Apr 28 '21 at 17:10
  • 1
    @nha I'm happy it works! I've written everything as an answer and tried better organize what I said xD – AndD Apr 28 '21 at 19:57

1 Answers1

2

As you want the annotation nginx.ingress.kubernetes.io/backend-protocol: "GRPC" to apply only on certain routes of your host, you could declare two Ingress definitions. The first one for all HTTP routes, the second one for GRPC routes.

The Nginx Ingress Controller will pick all the Ingress definitions (with the expected IngressClass) and will use them to compose the nginx.conf. This behaviour is perfect to have the possibility of having paths which requires different tunings in the annotations, like rewrite targets or, in your case, different backend protocols.

In particular, from the Nginx Controller documentation:

Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses.

You can check all the steps which are used to build the nginx.conf in the docs: https://kubernetes.github.io/ingress-nginx/how-it-works/#building-the-nginx-model

AndD
  • 2,281
  • 1
  • 8
  • 22