0

I have a Windows Server 2012 R2 IIS web server, running on VMWare esxi 5.5, hosting multiple websites with different IP addresses. This server recently has been refusing connections randomly across all websites. When the server stops accepting connections, there are tons of SYN_RECEIVED entries in netstat. Sometimes from the same IP address across all the hosted IP addresses other times from different IP addresses but a lot more of them. These entries sit for a while and then will disappear (assuming a timeout period is reached).

I've read this is probably a SYN flood attack, but that Windows is supposed to have a built in way of handling these attacks. I looked at this article, https://blogs.technet.microsoft.com/nettracer/2010/06/01/syn-attack-protection-on-windows-vista-windows-2008-windows-7-windows-2008-r2-windows-88-1-windows-2012-and-windows-2012-r2/, and did a netsh trace, but there's nothing in the logs that indicate SYN attack protection has been initiated. This makes me wonder whether SYN attack protection is working. The web server does have the built in Windows firewall disabled. Could that cause the SYN attack protection to be disabled?

I also read that increasing the server resources could help. I doubled the CPU and RAM for the virtual machine, but this problem still crops up every week. Usually for a day and then goes back to normal. When the server stops responding to connections, netstat will be filled with a bunch of SYN_RECEIVED entries and it will take several minutes to complete. At this time the CPU usage is really low, around 5-10%, and the RAM usage is half of what is available. After the suspected attack, netstat will return all the results in a few seconds.

How can I stop this connection refused thing from happening? Any tips or tricks? Does the disabled windows firewall have something to do with the mitigating the attacks.

awilinsk
  • 332
  • 4
  • 12
  • Check your routing and IPs. This can happen if (for example) something routes the packets back to the wrong host, or ARP tables are not updated because the requests end up on the wrong server, or or or. I had a similar problem caused by setting an incorrect configfile for a webserver-container, so that the routing to and from the container broke _sometimes_. Could be something along these lines. Also I would not disable the firewall, rather configure it correctly. – Lenniey Aug 15 '19 at 11:47

1 Answers1

1

I think this is difficult to stop at your Windows machine.

You should ensure you have a firewall, if not on the Windows host then certainly at the boundary.

I have successfully monitored connections on a Linux machine to identify unconventional behaviour like a SYN flood (which the linux kernel has some options for coping with, connecting from unusual ports (making connections from port 80/443 rather than to it for example), and so on, then flagging these and implementing a block in the firewall for the offending addresses.

On a Windows machine you could probably monitor the event log for information about this, and then manipulate the system firewall, but I think it's going to be harder and less reliable that using Linux where doing this fits with the kernel design.

If you wanted to try and do it in the kernel you would need to write a kernel mode driver to inspect packets, which is likley to be difficult and time consuming.

If you need to run your hosts under Windows, you might investigate whether you could setup a Linux machine which could filter the traffic and act as the network boundary.

And make sure that there is a firewall somewhere!

Rob Lambden
  • 260
  • 2
  • 6
  • I would hate to block the incoming IP addresses, because if it is a SYN flood attack, they're most likely spoofed IPs and I wouldn't want to block legitimate requests in the future. – awilinsk Aug 15 '19 at 12:08
  • But you need to protect yourself at that instant. In the Linux scenario I mentioned we created a number of IP Address lists in the kernel and added address to be blocked into those lists according to the time of day, and then periodically cleaned the lists so we could ensure any block was automatically removed after a period of time. – Rob Lambden Aug 15 '19 at 12:14
  • There's software for that, e.g. Fail2Ban – Lenniey Aug 15 '19 at 12:19
  • What software package would do that? – awilinsk Aug 15 '19 at 12:20
  • At the boundary (running Linux) we used standard iptables. For some scenarios we used a 'firewall mark' to pass the packets through userspace code to initiate the block, for others we added straight off the iptables rule. You can use the 'recent' check to look for addresses already known (in a kernel list) which we can then simply drop. – Rob Lambden Aug 15 '19 at 12:22
  • @Lenniey Fail2Ban is useful for application level checks (it wil parse log files), but a SYN flood never hits the application as it never completes the TCP handshake. – Rob Lambden Aug 15 '19 at 12:24
  • That's exactly what I'm experiencing. Even though the TCP connection is destined for one of our web servers on port 80 or 443, I never see this in any IIS logs because it never makes it to IIS. – awilinsk Aug 15 '19 at 12:33
  • @RobLambden yes, you either check this by rate controlling using e.g. iptables, or you write a log with iptables and parse that using e.g. F2B – Lenniey Aug 15 '19 at 12:37
  • Thanks all. I think I'm ultimately going to go the route that suggested here with a separate firewall filtering out this sort of traffic. – awilinsk Aug 16 '19 at 11:20