12

An nginx/1.0.12 running as a proxy on Debian 6.0.1 starts throwing the following error after running for a short time:

connect() to upstreamip:80 failed (99: Cannot assign requested address) 
while connecting to upstream, client: xxx.xxx.xxx.xxx, server: localhost, 
request: "GET / HTTP/1.1", upstream: "http://upstreamip:80/", 
host: "requesteddomain.com"

Not all requests produce this error, so I suspect that it has to do with the load of the server and some kind of limit it hit.

I have tried raising ulimit -n to 50k and worker_rlimit_nofile to 50k as well, but that does not seem to help. lsof -n shows a total of 1200 lines for nginx. Is there a system limit on outgoing connections that might prevent nginx from opening more connections to its upstream server?

mariow
  • 608
  • 1
  • 4
  • 12

4 Answers4

24

Seems like I just found the solution to my own question: Allocating more outgoing ports via

echo "10240 65535" > /proc/sys/net/ipv4/ip_local_port_range

solved the problem.

mariow
  • 608
  • 1
  • 4
  • 12
  • 5
    You can also make this change persistent by adding a `net.ipv4.ip_local_port_range = 10240 65535` line to `/etc/sysctl.conf` and calling `sudo sysctl -p` – Himura Sep 09 '20 at 13:55
3

Each TCP connection has to have a unique quadruple source_ip:source_port:dest_ip:dest_port

source_ip is hard to change, source_port is chosen from ip_local_port_range but can't be more than 16 bits. The other thing left to adjust is dest_ip and/or dest_port. So add some IP aliases for your upstream server:

upstream foo { server ip1:80; server ip2:80; server ip3:80; }

Where ip1, ip2 and ip3 are different IP addresses for the same server.

Or it might be easier to have your upstream listen on more ports.

Bryan Larsen
  • 9,468
  • 8
  • 56
  • 46
1

Accepted answer in a Docker compose:

version: "3.9"
services:
  nginx_proxy:
    build:
      context: ./src/proxy
    sysctls:
      net.ipv4.ip_local_port_range: "10240 65535"
David
  • 2,109
  • 1
  • 22
  • 27
0

modify /etc/sysctl.conf:

net.ipv4.tcp_timestamps=1
net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_max_tw_buckets=10000     #after done this: local ports decrease from 26000 to 6000(netstat -tuwanp | awk '{print $4}' | sort | uniq -c | wc -l)

run:

sysctl -p
diyism
  • 12,477
  • 5
  • 46
  • 46
  • I'm not sure if this would have helped because the problem wasn't a DOS attack resulting in lots of TIME_WAIT but just a huge number of regular traffic that was supposed to go through and not be killed with a faster TIME_WAIT timeout. – mariow Nov 11 '14 at 15:13
  • @mariow, on my server, there are huge quantity of outgoing requests(crawler), so fast TIME_WAIT reusing is essenstial. – diyism Dec 26 '14 at 02:20
  • 3
    net.ipv4.tcp_tw_recycle is broken and was removed from Linux 4.12: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4396e46187ca5070219b81773c4e65088dac50cc reuse is dangerous as well: https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux Better don't use these options. – pva Dec 05 '17 at 19:16