Nginx ingress controller websocket support

Question

Recently I've been working on a toy app using Kubernetes. Part of the app is a web server that needs to support WebSockets. Currently, I'm using port-forwarding to access the web server and everything works just fine.

I'd like to switch to using an Ingress and IngressController to avoid using the port forwarding.

Here is my Ingress config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
  rules:
  - http:
      paths:
      - path: /app
        backend:
          serviceName: web-svc
          servicePort: 3030
      - path: /ws
        backend:
          serviceName: web-svc
          servicePort: 3030

Now accessing the app through $(minikube ip)/app works just fine, but the WebSocket requests all fail because nginx is returning a 200 and not a 101.

I've tried adding the nginx.org/websocket-services annotation but that doesn't seem to be working either.

Has anyone encountered a similar situation?

Cheers

According to https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#websockets "Support for websockets is provided by NGINX out of the box. No special configuration required." But I can't get it to work. Also it is easy to get confused due to the two nginx controllers. See https://www.nginx.com/blog/wait-which-nginx-ingress-controller-kubernetes-am-i-using/ — Tom, Apr 27 '19 at 05:56
the quotes around the timeouts seem to be important with newer k8s versions — U.V., Mar 09 '21 at 16:59

Rico · Answer 1 · 2020-07-22T05:23:21.367

19

From looking at the nginx ingress controller docs and the nginx docs you probably need something like this as an annotation on your Kubernetes Ingress:

nginx.ingress.kubernetes.io/configuration-snippet: |
   proxy_http_version 1.1;
   proxy_set_header Upgrade "websocket";
   proxy_set_header Connection "Upgrade";

Note that once you add that annotation all of your Ingress rules will have that snippet in the location block in your nginx configs. So if you want to ignore it for other rules you will have to create a separate Kubernetes Ingress.

EDIT:

As per the gist and the Nginx ingress docs , it seems like this annotation fixed the problem:

nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600

edited Jul 22 '20 at 05:23

answered Oct 18 '18 at 23:51

Rico

58,485
12
111
141

I'm still fighting. I will comment back here when I understand the problem – Tom Apr 28 '19 at 03:38
2

Ok I did it. I have described the working configuration in this gist: https://gist.github.com/jsdevtom/7045c03c021ce46b08cb3f41db0d76da#file-ingress-service-yaml – Tom Apr 28 '19 at 06:04
@Tom can you describe what's the important part? I seem to be missing it from your gist. – VsMaX Jun 08 '20 at 17:59
I agree . @tom in your snippet I couldn't find nginx-ingress configuration snippet as to how did that work What you have is the ingress rule and not ingress controller annotation. – vgdub Jul 14 '20 at 17:53
Edited the question, seems like it was the timeout annotations – Rico Jul 22 '20 at 05:23

score 7 · Answer 2 · edited Feb 10 '23 at 12:08

7

Theese worked for me.

It seems they added support via annotation (example in docs):

nginx.org/websocket-services: "service1[,service2,...]"

I tested my connection with telsocket, small tool to connect to a WS/WSS socket.

There are all sorts of different clients, this might also help finding the culprit with the connection.

Important to note: two nginx ingress controllers are available, more info here.

This answer is limited to the nginxinc version, that is different that used in the question, the accepted answer is the only solution as of right now.

edited Feb 10 '23 at 12:08

Spikatrix

20,225
7
37
83

answered May 24 '21 at 19:06

NotMyDay

71
1
4

supported since which version ? – Abdennour TOUMI Sep 27 '21 at 06:15
Based on git history of the linked file, since v1.12.1. You can check the commit [here](https://github.com/nginxinc/kubernetes-ingress/commit/cad5d7ded7927d1994bfec43e2c06183295c23f4#diff-cc5dca25dead8470544fcfe6a9297e82de1b3897c092fb519e2019dd4c066d6e) . – NotMyDay Sep 28 '21 at 12:31
My bad, it is included since 2016, so the first tag contained this code was: v0.3. – NotMyDay Sep 28 '21 at 12:38
Does this only work with the NGINX Inc controller? – Aleksandr Dubinsky Dec 14 '21 at 10:55
Yes, unfortonutely it is limited to the nginxinc version. – NotMyDay Dec 15 '21 at 13:18

score 1 · Answer 3 · answered Mar 15 '22 at 09:36

The solution that I've used (please check the annotations area):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: tornado-socket
 annotations:
  kubernetes.io/ingress.class: nginx
  nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  nginx.ingress.kubernetes.io/server-snippets: |
   location / {
    proxy_set_header Upgrade $http_upgrade;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header Connection "upgrade";
    proxy_cache_bypass $http_upgrade;
    }
spec:....

It seems that this annotations are required for the free version of Ingress-NgInx. Full article: https://www.civo.com/learn/using-websockets-with-ingress-controller

score 1 · Answer 4 · answered Mar 22 '23 at 13:16

1

for people who are using classic ELB using nginx ingress controller . websocket is not supported. don't spend more time . https://aws.amazon.com/elasticloadbalancing/features/

answered Mar 22 '23 at 13:16

Arjun Dandagi

160
3
14

Nginx ingress controller websocket support

4 Answers4

Linked