I am using Express and HPM to proxy all requests to my website. This is all wrapped together into a little tool I call ws-proxy (ws for web server, not websocket).
One of the things proxied is my PVE/Proxmox Virtual Environment node, which uses secure WebSockets for the xterm.js and NoVNC consoles.
What is weird about this, is after starting ws-proxy, I have about 30 seconds to open a console which will be sustained, but connections after this time will be closed with a 404 Not Found error. In the console, I see
[HPM] Upgrading to WebSocket
[HPM] Upgrading to WebSocket (sometimes up to 4 times)
[HPM] Client disconnected
In my browser, I see the connection returned as 404.
With websocat, I get:
websocat: WebSocketError: Received unexpected status code (404 Not Found)
websocat: error running
After additional debugging, I see something in the stack is sending a 404 and closing the connection, where just afterwards PVE sends the 101 Switching Protocols. This also sometimes causes a write after end error, sometimes socket hangup.
I've spent months looking into this and I have nowhere else to look at this point.
http-proxy-middleware#826 (by me)
404 in inspect element:
error log in console after a recent attempt (error will change)
Full list of steps between client and server:
- Cloudflare
- DigitalOcean w/ ssh-forward (not the problem)
- ws-proxy
- server
Non-websocket (HTTP) requests work fine. This is with HPM v2 and Node.js v16.
Update 1 After Ryker's answer, I attempted the solution which should have fixed it, but I see something else of concern after setting the logLevel to debug:
0|ws-proxy | pve.internal.0xlogn.dev ::1 - - [02/Nov/2022:23:17:14 +0000] "POST /api2/json/nodes/proxmox/lxc/105/termproxy HTTP/1.1" 200 487 "https://pve.internal.0xlogn.dev/?console=lxc&vmid=105&node=proxmox&resize=scale&xtermjs=1" "Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0"
0|ws-proxy | Upgrade request for vhost pve.internal.0xlogn.dev, proxy out
0|ws-proxy | [HPM] GET /api2/json/nodes/proxmox/lxc/105/vncwebsocket?port=5900&vncticket=REDACTED -> https://10.0.1.2:8006
0|ws-proxy | [HPM] GET /api2/json/nodes/proxmox/lxc/105/vncwebsocket?port=5900&vncticket=REDACTED -> http://10.0.1.108:80
0|ws-proxy | [HPM] Upgrading to WebSocket
0|ws-proxy | [HPM] Upgrading to WebSocket
0|ws-proxy | [HPM] Client disconnected
0|ws-proxy | [HPM] GET /api2/json/cluster/resources -> https://10.0.1.2:8006
Notice the two GET requests? Something is duplicating the request.
My 'upgrade' event listener:
httpsServer.on('upgrade', (req, socket, head) => {
if (!req.headers.host) {
console.log('No vhost specified in upgrade request. Ignoring.');
socket.end();
return;
} else {
console.log(`Upgrade request for vhost ${req.headers.host}, proxy out`);
vhostProxyMiddlewareList[req.headers.host].upgrade(req, socket, head);
}
})
What's even weirder here, is after restarting, I get a short time where the request isn't duplicated. Plus, there is a normal HTTP request anyway.
Update 2 After noticing the dual requests, I believe it is possible the module vhost is causing a weird wildcard and sending the request to two target nodes. I will update shortly.
Update 3
After further work I believe this is true. However, vhost is not at fault, rather something is implicitly calling next().
Update 4 This is still an issue, even after multiple attempts at changing this. I have not heard anything back from HPM.
