Standalone WebSocket client throught Ingress nginx fails

Question

In a Hosted Rancher Kubernetes cluster, I have a service that exposes a websocket service (a Spring SockJS server). This service is exposed to the outside thanks an ingress rule:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myIngress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600s"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600s"
    nginx.ingress.kubernetes.io/enable-access-log: "true"
spec:
  rules:
    - http:
        paths:
        - path: /app1/mySvc/
          backend:
            serviceName: mySvc
            servicePort: 80

A web application connects to the web socket service throught an ingress nginx and it works fine. The loaded js scripts is:

    var socket = new SockJS('ws');
    stompClient = Stomp.over(socket);

    stompClient.connect({}, onConnected, onError);

On the contrary, the standalone clients (js or python) do not work as they returns a 400 http error.

For example, here is the request sent by curl and the response from nginx:

curl  --noproxy '*' --include \
     --no-buffer \
     -Lk \
--header "Sec-WebSocket-Key: l3ApADGCNFGSyFbo63yI1A==" \
--header "Sec-WebSocket-Version: 13" \
--header "Host: ingressHost" \
--header "Origin: ingressHost" \
--header "Connection: keep-alive, Upgrade" \
--header "Upgrade: websocket" \
--header "Sec-WebSocket-Extensions: permessage-deflate" \
--header "Sec-WebSocket-Protocol: v10.stomp, v11.stomp, v12.stomp" \
--header "Access-Control-Allow-Credentials: true" \
https://ingressHost/app1/mySvc/ws/websocket


HTTP/2 400
date: Wed, 20 Nov 2019 14:37:36 GMT
content-length: 34
vary: Origin
vary: Access-Control-Request-Method
vary: Access-Control-Request-Headers
access-control-allow-origin: ingressHost
access-control-allow-credentials: true
set-cookie: JSESSIONID=D0BC1540775544E34FFABA17D14C8898; Path=/; HttpOnly
strict-transport-security: max-age=15724800; includeSubDomains

Can "Upgrade" only to "WebSocket".

Why does it work with the browser and not standalone clients ?

Thanks

score 2 · Answer 1 · answered Nov 20 '19 at 20:54

The problem doesn't seem to be in the nginx Ingress. The presence of the JSESSIONID cookie most likely indicates that the Spring applications gets the request and sends a response.

A quick search through the Spring's code shows that Can "Upgrade" only to "WebSocket". is returned by AbstractHandshakeHandler.java when the Upgrade header isn't equal to WebSocket (case-insensitive match).

I'd suggest double-checking that the "Upgrade: websocket" header is present when making the call with curl.

Also, this appears to be a similar problem and may apply here as well if the application has several controllers.

And for what it's worth, after replacing ingressHost appropriately I tried the same curl command as in the question against https://echo.websocket.org and a local STOMP server I've implemented some time ago. It worked for both.

You may have already done this but have you tried capturing in the browser the Network traffic to see the request/response exchange, especially the one that returns HTTP 101 Switching Protocols? Then try to exactly replicate the call that the browser makes and that is successful. For example, the STOMP client generates a session id and uses a queue/topic, which are put in the URL path in requests to the server (e.g. /ws/329/dt1hvk2v/websocket). The test request with curl doesn't seem to have them.

score 1 · Answer 2 · answered Nov 20 '19 at 16:43

1

I had used nginx to create ingress until I discovered traefik. you should test with traefik.

answered Nov 20 '19 at 16:43

Antonio Fernandez

137
1
6

crepu · Answer 3 · 2019-11-21T11:08:09.260

1

I activated trace in Spring application and a call with curl gives:

o.s.w.s.s.s.DefaultHandshakeHandler      : Handshake failed due to invalid Upgrade header: null

So, it seems nginx removes the Upgrade header !

When testing with the browser client, the header is present in Spring Applicaton and it works.

edited Nov 21 '19 at 11:08

answered Nov 21 '19 at 10:49

crepu

91
9

score 1 · Answer 4 · answered Nov 25 '19 at 12:34

1

When I try with http instead of https, it works.

I do not understand why https removes Upgrade header for standalone clients.

answered Nov 25 '19 at 12:34

crepu

91
9

score 1 · Answer 5 · answered Nov 04 '20 at 12:22

I was facing the same issue with the headers not being forwarded. Decided to use the NGINX ingress controller provided by NGINX itself. It offers the nginx.org/websocket-services annotation that works as a breeze. You can read the configuration example here.

Here is the configuration that did the tricky for me.

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
 annotations:
   kubernetes.io/ingress.class: nginx
   nginx.org/websocket-services: "blue-svc"

Standalone WebSocket client throught Ingress nginx fails

5 Answers5