I'm trying to setup udp load balancer with IP and port transparent proxy. My environment runs on docker. We are using mesos + marathon to manage containers. For UDP load balancing I use Nginx. App and Nginx runs in docker containers. I have few requirements: 1. One client should always connect to the same app node during his session. 2. App behind nginx should assume that it is communicating to the client directly.
I used this article for my setup: https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/
So nginx is running with root user. Containers are on the same network. I've included a bunch of modules for nginx like stream and nginx-sticky-module-ng and etc.
Nginx Stream conf: `
upstream app-server {
sticky;
server some-app:5684;
server some-app:5684;
}
server {
listen 5684 udp;
proxy_pass app-server:5684;
proxy_bind $remote_addr:$remote_port transparent;
proxy_responses 1;
proxy_timeout 1s;
}
On Each upstream server I set default gw to Nginx IP.
On nginx I set following iptables rules.
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p udp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p udp -s 172.16.0.0/16 --sport 5684 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 0 #I assume this should be 0,because there could be different ports
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
But client can't reach the server it seems. It looks like packets just got stuck on the Nginx. Am I missing some configuration or something wrong with iptables rules?
