How do I get hasura websocket to work on my local Kubernetes cluster?

Question

I have set up a local K8s cluster under Windows like this:

install docker for desktop
in docker for desktop, enable kubernetes
install the nginx ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.29.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.29.0/deploy/static/provider/cloud-generic.yaml

add the following domain to the hosts (C:\Windows\System32\drivers\etc\hosts)

127.0.0.1  localhost api.shopozor

I'm doing nothing special here, I keep everything to the default setup.

Then, I deployed hasura to my cluster, with the following yamls (I am not showing the postgres deployments for the sake of conciseness):

---
# Source: api/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
type: Opaque
data:
  admin-secret: "c2VjcmV0"
---
# Source: api/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
      # TODO: we cannot use string port because devspace doesn't support it in its UI
      # targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: api
    app.kubernetes.io/instance: api
---
# Source: api/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: api
      app.kubernetes.io/instance: api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
        app.kubernetes.io/instance: api
    spec:
      serviceAccountName: api
      securityContext:
        {}
      initContainers:
      # App has to wait for the database to be online "depends_on" workaround
      - name: wait-for-db
        image: darthcabs/tiny-tools:1
        args:
        - /bin/bash
        - -c
        - >
          set -x;
          while [[ "$(nc -zv 'postgres' 5432 &> /dev/null; echo $?)" != 0 ]]; do
            echo '.'
            sleep 15;
          done
      containers:
      - name: api
        securityContext:
            {}
        image: shopozor/graphql-engine:EM5Aya
        imagePullPolicy: 
        env:
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: shared-postgresql
              key: postgresql-username
        - name: POSTGRES_DATABASE
          value: shopozor
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: shared-postgresql
              key: postgresql-password
        - name: POSTGRES_HOST
          value: postgres
        - name: POSTGRES_PORT
          value: "5432"
        - name: HASURA_GRAPHQL_SERVER_PORT
          value: "8080"
        - name: HASURA_GRAPHQL_ENABLE_CONSOLE
          value: "true"
        - name: HASURA_GRAPHQL_ENABLED_LOG_TYPES
          value: startup, http-log, webhook-log, websocket-log, query-log
        - name: HASURA_GRAPHQL_ENABLE_TELEMETRY
          value: "false"
        - name: HASURA_GRAPHQL_CORS_DOMAIN
          value: "*"
        - name: HASURA_GRAPHQL_DISABLE_CORS
          value: "false"
        - name: HASURA_GRAPHQL_UNAUTHORIZED_ROLE
          value: incognito
        - name: HASURA_GRAPHQL_ADMIN_SECRET
          valueFrom:
            secretKeyRef:
              name: api
              key: admin-secret
        - name: HASURA_GRAPHQL_JWT_SECRET
          value: "{\"type\": \"HS256\", \"key\": \"my-access-token-signing-key-secret\", \"audience\": [\"58640fbe-9a6c-11ea-bb37-0242ac130002\", \"6e707590-9a6c-11ea-bb37-0242ac130002\"], \"claims_namespace\": \"https://hasura.io/jwt/claims\", \"claims_format\": \"json\", \"issuer\": \"shopozor.com\" }"
        - name: HASURA_GRAPHQL_DATABASE_URL
          value: postgres://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@$(POSTGRES_HOST):$(POSTGRES_PORT)/$(POSTGRES_DATABASE)
        - name: FUNCTION_NAMESPACE
          value: dev
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: http
        readinessProbe:
          httpGet:
            path: /healthz
            port: http
        resources:
            {}
---
# Source: api/templates/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: "api.shopozor"
      http:
        paths:
          - path: /
            backend:
              serviceName: api
              servicePort: 8080

Now, I have a nuxt frontend that tries to make use of hasura's websocket. I configured apollo in the standard way

//---
// nuxt.config.js
[...]
  // Give apollo module options
  apollo: {
    cookieAttributes: {
      expires: 7
    },
    includeNodeModules: true, 
    authenticationType: 'Basic', 
    clientConfigs: {
      default: '~/apollo/clientConfig.js'
    }
  },
[...]
//---
// apollo/clientConfig.js
import { InMemoryCache } from 'apollo-cache-inmemory'
export default function (context) {
  return {
    httpLinkOptions: {
      uri: 'http://api.shopozor/v1/graphql',
      credentials: 'same-origin'
    },
    cache: new InMemoryCache(),
    wsEndpoint: 'ws://localhost:8080/v1/graphql'
  }
}

Notice I need no particular header currently. I should be able to access the websocket without authorization token.

Now, when I start my application, the websocket connection tries to initialize. If I port-forward my hasura service, then the above configuration is fine. The websocket connection seems to be working. At least, the hasura log shows

2020-07-16T06:49:59.937386882Z {"type":"websocket-log","timestamp":"2020-07-16T06:49:59.937+0000","level":"info","detail":{"event":{"type":"accepted"},"connection_info":{"websocket_id":"8437b784-1fce-4430-9ca9-a9e7517307f0","token_expiry":null,"msg":null},"user_vars":null}}

If I, however, change the wsEndpoint in the above configuration to use the ingress into my hasura instance,

wsEndpoint: 'ws://api.shopozor/v1/graphql'

then it doesn't work anymore. Instead, I continuously get a 404 Not Found. I can, however, access the hasura console through http://api.shopozor. The hasura log shows

2020-07-16T10:37:53.564657244Z {"type":"websocket-log","timestamp":"2020-07-16T10:37:53.564+0000","level":"error","detail":{"event":{"type":"rejected","detail":{"path":"$","error":"only '/v1/graphql', '/v1alpha1/graphql' are supported on websockets","code":"not-found"}},"connection_info":{"websocket_id":"5e031467-fb5c-460d-b2a5-11f1e21f22e7","token_expiry":null,"msg":null},"user_vars":null}}

So I googled a lot, found some information on annotations I should use in my ingresses, but nothing worked. What am I missing here? Do I need a particular nginx ingress controller configuration? Do I need to pass some special annotations to my hasura ingress? What do I need to do to make it work?

EDIT

To answer a question to this post, here's my the ingress to hasura I have applied on my cluster:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

EDIT 2

With the following ingress for my hasura instance

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_http_version 1.1;
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

the frontend application still has the same issue with websockets. In addition, it cannot really connect to hasura anymore. Instead, I get the following errors:

 ERROR  Network error: Unexpected token < in JSON at position 0                                                                                                                                         23:17:06

  at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1588:34
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2008:15
  at Set.forEach (<anonymous>)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2006:26
  at Map.forEach (<anonymous>)
  at QueryManager.broadcastQueries (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2004:20)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1483:29
  at runMicrotasks (<anonymous>)
  at processTicksAndRejections (internal/process/task_queues.js:97:5)

Global error handler                                                                                                                                                                                    23:17:06

 ERROR  Network error: Unexpected token < in JSON at position 0                                                                                                                                         23:17:06

  at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1486:27
  at runMicrotasks (<anonymous>)
  at processTicksAndRejections (internal/process/task_queues.js:97:5)

as well as

client.js?06a0:49 ApolloError: Network error: Unexpected token < in JSON at position 0
    at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
    at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1486:27
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  graphQLErrors: [],
  networkError: SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
      at JSON.parse (<anonymous>)
      at D:\workspace\shopozor\services\node_modules\apollo-link-http-common\lib\index.js:35:25
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (internal/process/task_queues.js:97:5) {
    name: 'ServerParseError',
    response: Body {
      url: 'http://api.shopozor/v1/graphql/',
      status: 404,
      statusText: 'Not Found',
      headers: [Headers],
      ok: false,
      body: [PassThrough],
      bodyUsed: true,
      size: 0,
      timeout: 0,
      _raw: [Array],
      _abort: false,
      _bytes: 153
    },
    statusCode: 404,
    bodyText: '<html>\r\n' +
      '<head><title>404 Not Found</title></head>\r\n' +
      '<body>\r\n' +
      '<center><h1>404 Not Found</h1></center>\r\n' +
      '<hr><center>nginx/1.17.8</center>\r\n' +
      '</body>\r\n' +
      '</html>\r\n'
  },
  message: 'Network error: Unexpected token < in JSON at position 0',
  extraInfo: undefined
}

and

index.js?a6d6:111 OPTIONS http://api.shopozor/v1/graphql/ net::ERR_ABORTED 404 (Not Found)

and

Access to fetch at 'http://api.shopozor/v1/graphql/' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Without the new annotations

nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_http_version 1.1;
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";

on my hasura ingress, the connection between my frontend and hasura is working fine, except for the websockets.

EDIT 3

I tried the following two ingresses, with no success:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_http_version 1.1;
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

and

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

In the latter case, I just get the error

WebSocket connection to 'ws://api.shopozor/v1/graphql/' failed: Error during WebSocket handshake: Unexpected response code: 404

while the graphql endpoint is functional. In the former case, I can't access to anything on the hasura instance and I get the websocket handshake issue above (so no graphql and no websocket working).

EDIT 4

With my api ingress configuration (without any additional nginx annotations like the above: nginx.ingress.kubernetes.io/proxy-read-timeout, nginx.ingress.kubernetes.io/proxy-send-timeout, nginx.ingress.kubernetes.io/configuration-snippet), if I do this:

curl -i -N -H "Connection: Upgrade" \
 -H "Upgrade: websocket" \
 -H "Origin: http://localhost:3000" \
 -H "Host: api.shopozor" \ 
 -H "Sec-Websocket-Version: 13" \
 -H "Sec-WebSocket-Key: B8KgbaRLCMNCREjE5Kvg1w==" \ 
 -H "Sec-WebSocket-Protocol: graphql-ws" \
 -H "Accept-Encoding: gzip, deflate" \
 -H "Accept-Language: en-US,en;q=0.9" \
 -H "Cache-Control: no-cache" \
 -H "Pragma: no-cache" \
 -H "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits" \
 -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.49 Safari/537.36" \
 http://api.shopozor/v1/graphql

then hasura is happy:

2020-07-28T07:42:28.903877263Z {"type":"websocket-log","timestamp":"2020-07-28T07:42:28.894+0000","level":"info","detail":{"event":{"type":"accepted"},"connection_info":{"websocket_id":"94243bde-41c4-42c8-8d8f-355c47a3492e","token_expiry":null,"msg":null},"user_vars":null}}

The headers in my curl above are the very same headers as those sent by my frontend app. Any clue on what I am doing wrong? The difference between the frontend calls and this curl is that in the frontend I define the websocket endpoint to be ws://api.shopozor/v1/graphql while I curl http://api.shopozor/v1/graphql. I cannot, in apollo vue, set the wsEndpoint to http://api.shopozor/v1/graphql. I get an error.

The hint is the `"path":"$"` in the error log. Check your ingress for misconfig. Share your ingress manifest here. — Lukman, Jul 16 '20 at 12:57

Rico · Answer 1 · 2020-07-22T05:21:33.727

4

Sounds like you are using a nginx ingress controller and WebSockets are supported out of the box.

You can try this annotation :

nginx.ingress.kubernetes.io/configuration-snippet: |
   proxy_http_version 1.1;
   proxy_set_header Upgrade "websocket";
   proxy_set_header Connection "Upgrade";

or/and this other annotation, since they docs recommend to use that for WebSockets:

nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600

Note: I answered a similar question a while ago ⌛⌚.

edited Jul 22 '20 at 05:21

answered Jul 17 '20 at 04:26

Rico

58,485
12
111
141

I saw that suggestion somewhere, I tried it out, I probably misunderstood it, because it did not work. Where should that annotation go? in the ingress of my hasura instance? – Laurent Michel Jul 17 '20 at 05:26
yes... you are using an nginx ingress controller right? – Rico Jul 17 '20 at 05:28
Please see my "Edit 2" for more information on what is happening when I apply the annotations you propose. I must be doing something very wrong, but I can't figure out what. – Laurent Michel Jul 21 '20 at 21:25
Can you try the timeout annotations without the first configuration-snippet annotation? – Rico Jul 22 '20 at 05:24
I tried that and, as reported in my EDIT 3, it's not working either ... – Laurent Michel Jul 22 '20 at 18:29
what happens when EDIT 3? websockets don't work and graphql works? – Rico Jul 22 '20 at 18:40
without the `configuration-snippet`, yes, the websockets don't work and graphql works; with the `configuration-snippet`, none of the websockets and the graphql work – Laurent Michel Jul 22 '20 at 20:08
should the hasura service be of some other type than `ClusterIP`? does it have to be `LoadBalancer` for example? – Laurent Michel Jul 22 '20 at 20:09
`api` service? ClusterIP should be fine. K8s internal services are L4, you have an L7 problem. – Rico Jul 22 '20 at 20:25
I've just tried something else. I supposed that my cluster configuration is fine as it is and I tried to curl the websocket somehow. The result is listed in my EDIT 4. Any advice? – Laurent Michel Jul 28 '20 at 07:51
You can see what the actual header that is causing the problem by removing them one by one in your curl command. Then you can try adding that custom header in Nginx with https://www.keycdn.com/support/nginx-add_header. and configuration snippet – Rico Jul 28 '20 at 12:30
In my curl command, there is no problem at all, that's my concern. If I curl with the very same headers as my frontend app, then it works. – Laurent Michel Jul 28 '20 at 13:19
hmmm... yeah, it's a header missing/additional issue. – Rico Jul 28 '20 at 13:52

How do I get hasura websocket to work on my local Kubernetes cluster?

EDIT

EDIT 2

EDIT 3

EDIT 4

1 Answers1