13

We are regularly experiencing strange issues with networking on our dedicated server. It runs Windows Server 2012 R2 x64 on Xeon E5620 with 16 GB RAM and Intel 82575EB network adapter.

Please note that we've already tuned HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters key values TcpTimedWaitDelay and MaxUserPort to 30 and 65530 respectively.

At a random point of time our websites stop responding, the reason being they cannot connect to a local database. It's approximately 2 weeks uptime when this issues start taking place. The system log starts getting TCPIP warnings 4227 and 4231.It states "A request to allocate an ephemeral port number from the global TCP port space has failed due to all such ports being in use.".

If I run

Get-Counter -Counter \TCPv4\*

or

Get-Counter -Counter \TCPv6\*

or

netstat -abn | find /c ":"

I always get reasonable value of 500-1500 connections, which is not even close to 65K limit.

Additionally, "localhost" stops resolving to ::1 locally, reverting to 127.0.0.1
Only a forced machine restart can resolve the situation.

Could it be a network adapter problem?

UPDATE 1

It happened again and seemed to have been resolved when I restarted the mail server. Weird though, all the counters showed ~1000 connections with ~500 being active at the moment, and still the 10055 socket error when trying to connect to the database which has nothing to do with the mail server.

UPDATE 2 This IS strange, but the daily restart of the mail services fixes the problem completely.

CamaroSS
  • 243
  • 1
  • 3
  • 9
  • 1
    I had a similar problem and the following link fixed it for me: [http://blogs.technet.com/b/kimberj/archive/2012/07/06/sever-quot-hangs-quot-and-ephemeral-port-exhaustion-issues.aspx](http://blogs.technet.com/b/kimberj/archive/2012/07/06/sever-quot-hangs-quot-and-ephemeral-port-exhaustion-issues.aspx) –  Jul 21 '15 at 22:33
  • 2
    About UPDATE 2. So this simply indicates that mail services generate too many connections without closing them and you not solved problem but just hide it till it become worse and 2 restarts per day will be required some time... Looks like hiding rather than solving problem... – Mikhail Feb 14 '18 at 11:59

4 Answers4

9

I've had similar problem with exhausted pool of TCP/IP ports on WinSvr 2012R2 x64 for almost 1 month where server stopped receiving any new and TCP connections. So I played with registry values and these are stable for me:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"TcpTimedWaitDelay"=dword:0000001e
"MaxUserPort"=dword:0000fffe
"TcpNumConnections"=dword:00fffffe
"TcpMaxDataRetransmissions"=dword:00000005
  • TcpTimedWaitDelay - 30
  • MaxUserPort - 65534
  • TcpNumConnections - should be in default state stretched to maximum = 16777214 should prevent server from exhausting ephemeral ports.
  • TcpMaxDataRetransmissions - Timeout limit of TCP unacknowledged data segments retransmission on actual connection = 5.

In result same like yours. I think you should consider to audit your behavior of your applications/scripts performance. If everything is ok and nothing help, then you can try to put proxy server before your web application server, make 2 nodes with web server (IIS, Apache, ...) which will share same static content and access same database at the same time (if you have enough resources in your company).

Maybe this article would help you in some way: http://blogs.technet.com/b/tristank/archive/2008/03/11/maxuserport-what-it-is-what-it-does-when-it-s-important.aspx

MyKE
  • 301
  • 2
  • 9
  • 1
    You should've read the question thoroughly before answering. Of course I did try this solution before asking, it's very common. I've already solved the problem by another means. – CamaroSS Jan 12 '15 at 13:08
  • 1
    I think daily restart of the mail services is not solved problem. Have you found another solution? – MyKE Jan 12 '15 at 13:22
  • 4
    @CamaroSS: Please share your solution (if it goes beyond "restarting the mail service" - this is *not* a solution). Also, maybe you should appreciate it a bit more if someone tries to help you. – Sven Jan 12 '15 at 13:22
  • @CamaroSS And if you read carefuly my answer you'll see "In result same like yours", then I've posted another information.. – MyKE Jan 12 '15 at 13:23
  • Unfortunately, this *is* a solution, at least for now. There is no comment from the mail server (Eserv) devs so far. I can't find a logical explanation to this, but it works. Maybe the mail server was leaking connections, but why were the system counters lying to me then? – CamaroSS Jan 12 '15 at 13:28
  • 1
    Does setting `MaxUserPort` still work in 2012? I thought in 2012 you must do it via netsh. e.g.: `netsh int ipv4 set dynamicport tcp start=25535 num=40000` – rustyx Oct 20 '15 at 12:19
5

Had this same issue on Windows Server 2016 running large amounts of Selenium tests using chromewebdriver. This PS script will automatically configure the settings @Myke shared above. The shutdown command was added because a reboot is required for the TCP stack changes.

Increase Pool Size for Ephemeral TCP Ports

Get-Item 'HKLM:\System\CurrentControlSet\Services\Tcpip\Parameters' | New-ItemProperty -Name MaxUserPort -Value 65534 -Force | Out-Null
Get-Item 'HKLM:\System\CurrentControlSet\Services\Tcpip\Parameters' | New-ItemProperty -Name TcpTimedWaitDelay -Value 30 -Force | Out-Null
Get-Item 'HKLM:\System\CurrentControlSet\Services\Tcpip\Parameters' | New-ItemProperty -Name TcpNumConnections -Value 16777214 -Force | Out-Null
Get-Item 'HKLM:\System\CurrentControlSet\Services\Tcpip\Parameters' | New-ItemProperty -Name TcpMaxDataRetransmissions -Value 5 -Force | Out-Null

shutdown -r -t 0

This was the error message we were receiving on Webdriver.Quit() telling us that a TCP address was in use.

Error: EADDRINUSE connect EADDRINUSE 127.0.0.1:12843 at ClientRequest. (\node_modules\selenium-webdriver\http\index.js:238:15)
From: Task: WebDriver.quit()

SliverNinja - MSFT
  • 173
  • 2
  • 8
5

In addition to the Tcpip driver settings, the ephemeral TCP port range is managed in Windows Server using the netsh command (source).

You can view the dynamic port range with the following commands:

  • netsh int ipv4 show dynamicport tcp
  • netsh int ipv4 show dynamicport udp
  • netsh int ipv6 show dynamicport tcp
  • netsh int ipv6 show dynamicport udp

To change the port range, use this command:

  • netsh int <ipv4|ipv6> set dynamic <tcp|udp> start=number num=range

For example:

netsh int ipv4 set dynamicport tcp start=49152 num=16384

The setting (start=49152 num=16384) is also the default on Windows Server 2008 onwards.

rustyx
  • 1,676
  • 3
  • 21
  • 30
  • Thank you so much for this - I've already spent hours on trying to solve my problem. All the advice I could find was about adjusting MaxUserPort in the registry and nobody mentioned netsh. – milosz Aug 08 '17 at 07:14
1

Have you made sure that you are not leaking database connection objects? You have to close every database connection that is opened, either explicitly (with try-finally) or with a using { } block. This is a common problem that ASP won't directly tell you about.

James
  • 363
  • 2
  • 4
  • 16
  • It's a bunch of PHP sites, most of them are running via FastCGI using persistent connection, so it shouldn't be the case. If it was, then the system counters would return way higher values. I also can't explain why localhost suddenly stops to resolve to ::1 and resolves to 127.0.0.1 instead. – CamaroSS Dec 14 '14 at 07:56
  • 2
    ::1 disappearing is probably just a side effect of overflowing the allowable ports--I suspect that windows tries an IPv6 ping and when it can't get a port to make the attempt, it falls back to IPv4. PHP is outside my areas of expertise. :-(. Good luck! – James Dec 15 '14 at 03:11