Can't access websocket (ws://) url using marathon-lb

Question

I have a container running a jupyter gateway which requires two urls to be accessible a http and websocket url.

On localhost, for example, these urls are http://127.0.0.1:8888 and ws://127.0.0.1:8888.

When I start my app with marathon and ssh into the mesos slave running the container I can access both of these urls functionality using my client that makes requests to the jupyter gateway. Which tells me the jupyter gateway in the container is working fine.

However when I am trying to access the jupyter gateway through the marathon load balancer my client tells me it can reach the http:// url fine but it times out when trying to connect to the ws:// url which I need to be able to access.

I believe marathon-lb supports forwarding websockets without extra configuration so I am not sure what the issue is but I suspect it could do with the following haproxy config in my marathon app labels.

Labels:

HAPROXY_0_MODE=http
HAPROXY_0_PATH=/jupyter-gateway-container-path
HAPROXY_0_VHOST=ourwebsite.com

EDIT to add more logs:

Below are the client and server logs. The server is the Jupyter gateway. The client is the python client they provide in the examples repository. You can see that a https:// request goes through successfully from the client and it shows a 201 POST in the server's log. But the ws:// request times out.

Client log https:// request is fine but ws:// times out.

  $ python client.py
https://<websitename>.io/7c5e1967-b8e4-4e6d-bb68-80881d7f3de1
ws://<websitename>.io/7c5e1967-b8e4-4e6d-bb68-80881d7f3de1
Created kernel 11e9b48b-d0b9-4419-b13a-205eaee7f2c7. Connect other clients with the following command:
            docker-compose run client --kernel-id=11e9b48b-d0b9-4419-b13a-205eaee7f2c7

ws://<websitename>.io/7c5e1967-b8e4-4e6d-bb68-80881d7f3de1/api/kernels/11e9b48b-d0b9-4419-b13a-205eaee7f2c7/channels
Traceback (most recent call last):
  File "client.py", line 178, in <module>
    IOLoop.current().run_sync(main)
  File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 453, in run_sync
    return future_cell[0].result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "client.py", line 125, in main
    ws = yield websocket_connect(ws_req)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
tornado.httpclient.HTTPError: HTTP 599: Timeout

Server log recieves https request ok, no mention of further connections.

[KernelGatewayApp] Jupyter Kernel Gateway at http://0.0.0.0:8888
[KernelGatewayApp] Kernel started: 11e9b48b-d0b9-4419-b13a-205eaee7f2c7
[I 170519 01:12:22 web:1971] 201 POST /7c5e1967-b8e4-4e6d-bb68-80881d7f3de1/api/kernels (10.0.0.173) 35.77ms

Can you attach some logs from client and server? Have you seen [websockets load balancing with haproxy](https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/) — janisz, May 17 '17 at 11:25
I have attached the client and server logs. That post seems relevant but I am working with marathon-lb which generates the haproxy confing automatically. So I would need to understand how to configure marathon-lb to configure haproxy to support proxying websocket connections. It's also worth noting I can use the container locally with only one port open 8888 so I am not sure if that means both http and ws connections are going over that one port. — Hamzeh Alsalhi, May 19 '17 at 01:22

score 0 · Answer 1 · answered Nov 03 '17 at 22:14

Most likely you ran into an issue of having the correct check for the ports therefore you'll need something like the following:

"HAPROXY_0_BACKEND_HTTP_HEALTHCHECK_OPTIONS": "  option  httpchk GET {healthCheckPath}
 HTTP/1.1\\r\\nHost:\\ www\n
  timeout check {healthCheckTimeoutSeconds}s\n",
"HAPROXY_1_BACKEND_HTTP_HEALTHCHECK_OPTIONS": "  option  httpchk GET {healthCheckPath}
 HTTP/1.1\\r\\nHost:\\ www\n
  timeout check {healthCheckTimeoutSeconds}s\n",

Notice the HAPROXY_0 and HAPROXY_1 you'll need to configure it for each defined port in your application

To have the websockets actually running you'll need to make sure that the connection is identified:

acl is_websocket hdr(Upgrade) -i WebSocket
use_backend {backend} if is_websocket

In the context of your application you'll need to configure it like the following:

"HAPROXY_HTTP_FRONTEND_ACL": "  acl host_{cleanedUpHostname} hdr(host) -i {hostname}\\r\\n
  use_backend {backend} if host_{cleanedUpHostname}\\r\\n
  acl is_websocket hdr(Upgrade) -i WebSocket\\r\\n
  use_backend {backend} if is_websocket"

Also note here, I made sure the default configuration is still available so everything else works as the default but also with websockets.

so I have `"HAPROXY_0_HTTP_FRONTEND_ACL": "acl host_{cleanedUpHostname} hdr(host) -i {hostname}\\\\r\\\\n use_backend {backend} if host_{cleanedUpHostname}\\\\r\\\\n acl is_websocket hdr(Upgrade) -i WebSocket\\\\r\\\\n use_backend {backend} if is_websocket"` this label but it makes no changes in the haproxy config file. — Rohit Hazra, Apr 24 '19 at 08:28

Can't access websocket (ws://) url using marathon-lb

1 Answers1

Linked