Having just setup a project that is essentially identical to what you describe, I'll share my approach - no guarantees that it is 'the best', but it does work.
My server stack is
- Varnish (v3.0.2) - all interfaces, port 80
- Nginx (v1.0.14) - local interface, port 81
- Node.js (v0.6.13) - local interface, port 1337
- Operating system is CentOS 6.2 (or similar)
My Node.js app uses Websockets (sockets.io - v0.9.0) and Express (v2.5.8) - and is launched using forever. (The same server also has other sites on it - primarily PHP which use the same instances of Nginx and Varnish).
The basic intention of my approach is as follows:
- Single public port/address for both websocket and 'regular' data
- Cache some assets using Varnish
- Serve (uncached) static assets directly from nginx
- Pass requests for 'web pages' to nginx, and from their proxy to Node.js
- Pass web socket requests directly (from Varnish) to Node.js (bypass nginx).
Varnish config - /etc/varnish/default.vcl:
#Nginx - on port 81
backend default {
.host = "127.0.0.1";
.port = "81";
.connect_timeout = 5s;
.first_byte_timeout = 30s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
#Node.js - on port 1337
backend nodejs{
.host = "127.0.0.1";
.port = "1337";
.connect_timeout = 1s;
.first_byte_timeout = 2s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
sub vcl_recv {
set req.backend = default;
#Keeping the IP addresses correct for my logs
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
#remove port, if included, to normalize host
set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
#Part of the standard Varnish config
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.request != "GET" && req.request != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
#Taken from the Varnish help on dealing with Websockets - pipe directly to Node.js
if (req.http.Upgrade ~ "(?i)websocket") {
set req.backend = nodejs;
return (pipe);
}
###Removed some cookie manipulation and compression settings##
if(req.http.Host ~"^(www\.)?example.com"){
#Removed some redirects and host normalization
#Requests made to this path, even if XHR polling still benefit from piping - pass does not seem to work
if (req.url ~ "^/socket.io/") {
set req.backend = nodejs;
return (pipe);
}
#I have a bunch of other sites which get included here, each in its own block
}elseif (req.http.Host ~ "^(www\.)?othersite.tld"){
#...
}
#Part of the standard Varnish config
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
#Everything else, lookup
return (lookup);
}
sub vcl_pipe {
#Need to copy the upgrade for websockets to work
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
}
set bereq.http.Connection = "close";
return (pipe);
}
#All other functions should be fine unmodified (for basic functionality - most of mine are altered to my purposes; I find that adding a grace period, in particular, helps.
Nginx config - /etc/nginx/*/example.com.conf:
server {
listen *:81;
server_name example.com www.example.com static.example.com;
root /var/www/example.com/web;
error_log /var/log/nginx/example.com/error.log info;
access_log /var/log/nginx/example.com/access.log timed;
#removed error page setup
#home page
location = / {
proxy_pass http://node_js;
}
#everything else
location / {
try_files $uri $uri/ @proxy;
}
location @proxy{
proxy_pass http://node_js;
}
#removed some standard settings I use
}
upstream node_js {
server 127.0.0.1:1337;
server 127.0.0.1:1337;
}
I am not particularly crazy about the repetition of the proxy_pass statement, but haven't gotten around to finding a cleaner alternative yet, unfortunately. One approach may be to have a location block specifying the static file extensions explicitly and leave the proxy_pass statement outside of any location block.
A few settings from /etc/nginx/nginx.conf:
set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;
log_format timed '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'$request_time $upstream_response_time $pipe';
port_in_redirect off;
Among my other server blocks and settings, I also have gzip and keepalive enabled in my nginx config. (As an aside, I believe there is a TCP module for Nginx which would enable the use of websockets - however, I like using 'vanilla' versions of software (and their associated repositories), so that wasn't really an option for me).
A previous version of this setup resulted in an unusual 'blocking' behaviour with the piping in Varnish. Essentially, once a piped socket connection was established, the next request would be delayed until the pipe timed out (up to 60s). I haven't yet seen the same recur with this setup - but would be interested to know if you see a similar behaviour.
