How to serve websocket apps from inside lxc container?

Question

Inside lxc container I'm running a faye app.

Gemfile:

source 'https://rubygems.org'
gem 'faye'
gem 'thin'

config.ru:

require 'faye'
Faye::WebSocket.load_adapter('thin')
bayeux = Faye::RackAdapter.new(:mount => '/faye', :timeout => 25)
run bayeux

Then

$ thin start

/etc/nginx/sites-available/domain.com:

server {
    server_name   domain.com;
    location /faye {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
    }
}

On the host:

/etc/nginx/sites-available/domain.com:

server {
    server_name   domain.com;
    location / {
        proxy_pass   http://10.0.0.109:80;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Real-IP   $remote_addr;
        proxy_set_header   Host   $http_host;
    }
}

Then, I try to connect to it from here (ws://domain.com/faye), but it fails. What am I doing wrong?

The app says:

Using rack adapter
Thin web server (v1.6.4 codename Gob Bluth)
Maximum connections set to 1024
Listening on 0.0.0.0:3000, CTRL+C to stop

Guest nginx access log:

10.0.0.1 - - [09/Dec/2015:11:03:21 +0200] "GET /faye?encoding=text HTTP/1.0" 400 11 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

Host nginx access log:

11.111.111.111 - - [09/Dec/2015:11:03:21 +0200] "GET /faye?encoding=text HTTP/1.1" 400 21 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

chrome's Developer tools's console:

WebSocket connection to 'ws://domain.com/faye?encoding=text' failed: Error during WebSocket handshake: Unexpected response code: 400

I tried to run the app suggested by Myst on the guest.

Gemfile:

source 'https://rubygems.org'
gem 'plezi'

app.rb:

#!/usr/bin/env ruby
# require the gems
require 'bundler'
Bundler.require(:default, ENV['ENV'].to_s.to_sym)
# handle requests
class MyController
    # Http
    def index
        Iodine.log request_data_string
    end
    # Websockets
    def on_message data
        write ERB::Util.html_escape(data)
    end
    def pre_connect
        puts Iodine.log(request_data_string)
        true
    end
    def on_open
        write 'Welcome!'
    end
    # formatting the request data
    protected
    def request_data_string
        out = String.new
        out << "Request headers:\n"
        out << (request.headers.to_a.map {|p| p.join ': '} .join "\n")
        out << "\n\nRequest cookies:\n"
        out << (request.cookies.to_a.map {|p| p.join ': '} .join "\n")
        out << "\n\nAll request data:\n"
        out << (request.to_a.map {|p| p.join ': '} .join "\n")
        out
    end
end
route '*', MyController
# # you can also set up logging to a file:
# Plezi.logger = Logger.new("filename.log")

Then

$ ruby app.rb

/etc/nginx/sites-available/domain.com:

server {
    server_name   domain.com;
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
    }
}

With this when I connect to ws://domain.com/ the app says:

Iodine 0.1.19 is listening on port 3000
Plezi is feeling optimistic running version 0.12.21.

Press ^C to stop the server.
Request headers:
connection: upgrade
host: localhost:3000
x-forwarded-for: 11.111.111.111
pragma: no-cache
cache-control: no-cache
origin: http://www.websocket.org
sec-websocket-version: 13
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
accept-encoding: gzip, deflate, sdch
accept-language: en-US,en;q=0.8
sec-websocket-key: Qs2LMnJ12SjclOxlrYKwlg==
sec-websocket-extensions: permessage-deflate; client_max_window_bits

Request cookies:


All request data:
io: #<Iodine::Http::Http1:0x007fd5a0006b08>
cookies: {}
params: {:encoding=>"text"}
method: GET
query: /?encoding=text
version: 1.1
time_recieved: 2015-12-09 11:24:16 +0200
connection: upgrade
host: localhost:3000
x-forwarded-for: 11.111.111.111
pragma: no-cache
cache-control: no-cache
origin: http://www.websocket.org
sec-websocket-version: 13
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
accept-encoding: gzip, deflate, sdch
accept-language: en-US,en;q=0.8
sec-websocket-key: Qs2LMnJ12SjclOxlrYKwlg==
sec-websocket-extensions: permessage-deflate; client_max_window_bits
headers_complete: true
client_ip: 11.111.111.111
scheme: http
host_name: localhost
port: 3000
path:
original_path: /
query_params: encoding=text
host_settings: {:index_file=>"index.html", :assets_public=>"/assets", :public=>nil, :assets_public_regex=>/^\/assets\//i, :assets_public_length=>8, :assets_refuse_templates=>/(erb|coffee|scss|sass|\.\.\/)$/i}11.111.111.111 [2015-12-09 09:24:16 UTC] "GET / http/1.1" 200 1659 0.6ms

Guest nginx access log:

10.0.0.1 - - [09/Dec/2015:10:55:44 +0200] "GET /?encoding=text HTTP/1.0" 200 1526 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

Host nginx access log:

11.111.111.111 - - [09/Dec/2015:10:55:44 +0200] "GET /?encoding=text HTTP/1.1" 200 1526 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

chrome's Developer tools's console:

WebSocket connection to 'ws://domain.com/?encoding=text' failed: Error during WebSocket handshake: Unexpected response code: 200

I'm not sure, but I think Faye requires a specific javascript client and I think there might be some connection headers involved and an XHR request made before the websocket one (but I might be wrong)... — Myst, Dec 09 '15 at 04:39
Using `faye` was the easiest way to set up websocket server I knew. And when moved to host, I can connect to the app from [the page I mentioned](http://www.websocket.org/echo.html). — x-yuri, Dec 09 '15 at 08:34
So it's not a client issue... that's good to know. It might be the request headers from the nginx proxy. Did you try using [Plezi](http://www.plezi.io) to print out the headers to STDOUT or a log file? I posted a small script that will print out the headers to STDOUT... maybe it would help with finding the problem. — Myst, Dec 09 '15 at 09:00
Okay... my bad (sorry)... Please see the updated `index` method in my answer (notice I changed the `request_data` method name)... the index method should now be a single line showing `Iodine.log request_data`... The reason was that the Http path didn't log the headers, and since the issue is with the headers, the request was assumed to be Http instead of Websockets. — Myst, Dec 09 '15 at 09:20
Your missing the `upgrade: websocket` header... I have no idea why... try bumping the `/faye` setup so it's above the root setup (just a thought, but I'm at a loss for now, as it works on my machine, see my updated answer). — Myst, Dec 09 '15 at 09:54

Myst · Answer 1 · 2015-12-09T09:53:39.507

This is not an answer, just a thought I had about how to debug the issue.

Outputting the request's headers to a log will provide more information about the issue (the headers, as seen by the app, after the proxy header changes, might be different than expected).

Here's a small app using Plezi that is both a websocket Echo server and should print out the headers to a log (the default log is STDOUT, but you can use a file too), allowing you to review the nginx setup and request headers:

the gemfile:

gem plezi

It's important not to use thin or any other server, since Plezi uses it's own Iodine server.

the app.rb file:

#!/usr/bin/env ruby
# require the gems
require 'bundler'
Bundler.require(:default, ENV['ENV'].to_s.to_sym)
# handle requests
class MyController
    # Http
    def index
        Iodine.log request_data_string
    end
    # Websockets
    def on_message data
        write ERB::Util.html_escape(data)
    end
    def pre_connect
        puts Iodine.log(request_data_string)
        true
    end
    def on_open
        write 'Welcome!'
    end
    # formatting the request data
    protected
    def request_data_string
        out = String.new
        out << "Request headers:\n"
        out << (request.headers.to_a.map {|p| p.join ': '} .join "\n")
        out << "\n\nRequest cookies:\n"
        out << (request.cookies.to_a.map {|p| p.join ': '} .join "\n")
        out << "\n\nAll request data:\n"
        out << (request.to_a.map {|p| p.join ': '} .join "\n")
        out
    end
end
route '*', MyController
# # you can also set up logging to a file:
# Plezi.logger = Logger.new("filename.log")

to start:

ruby app.rb

or (to use any port number, 3000 is default):

./app.rb -p 3000

This should be a good enough app for testing and debugging any headers.

Post the header data in your question for any further help. It could be that the headers aren't correctly parsed for websocket connections.

EDIT

I noticed the response Plezi gave was a 200 OK status code - meaning the request was assumed to be an Http request.

This means it is probably a header's issue, as the request wasn't recognized to be a websocket upgrade request.

EDIT 2

On my machine, the following setup allows me to proxy both Http requests and Websocket requests (but I'm using Plezi, which lets me use both Websockets, Plezi's Http and a Rack application (i.e. Rails) on the same port):

    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $http_connection;
    }

... but your configuration worked too (just messed-up Http's keep-alive, not the websockets)...

... so I doubt if I found the issue.

I've figured it out. It didn't come to me to pass `Host` header. See [my answer](http://stackoverflow.com/a/34190367/52499). Thanks for bearing with me. — x-yuri, Dec 09 '15 at 22:41

score 0 · Accepted Answer · answered Dec 09 '15 at 22:40

Probably, the easier way to set it up is to proxy from host nginx directly to the app:

/etc/nginx/sites-available/domain.com (host):

server {
    server_name   domain.com;
    location / {
        proxy_pass   http://10.0.0.109:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
    }
}

But if you want to have the same nginx configuration regardless whether the app is running on the host, or guest, you can proxy it with two nginx.

/etc/nginx/sites-available/domain.com (host):

server {
    server_name   domain.com;
    location / {
        proxy_pass   http://10.0.0.109:80;
        proxy_set_header   Host   $http_host;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
    }
}

Passing Host header here is vital. Or else guest nginx might let the wrong virtual host handle the request. That was what I couldn't realize for quite a while.

/etc/nginx/sites-available/domain.com (guest):

server {
    server_name   domain.com;
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
    }
}

How to serve websocket apps from inside lxc container?

2 Answers2

EDIT

EDIT 2