How to deal with network port abuse in sockets

Question

I have a web app written in c++ using TCP/IP over standard network sockets, running on Linux. The service is open to the wild and woolly internet.

Periodically I get bursts of abusive requests from spammers running automated scripts. I can detect these and close down the socket. Right now I just do a polite socket close like I would do to any valid request that has completed, with a socket lib close like this:

close( mSocket );

But sometimes closing the socket normally notifies the spam script that the socket connection has terminated, and they immediately initiate another fraudulent request.

What is the best way to terminate a TCP/IP connection that cleans up the open socket on my system, but leaves the remote party hanging. That is I want to close a socket in the way that is lowest cost for me, but highest cost for them.

@Nicholas Wilson:

Using TCP_REPAIR seems like a good idea. When a socket is closed in TCP_REPAIR mode no FIN or RST packet is sent. The remote socket is left hanging. I'm going to try it and report back. Here is my (untested) code:

if ( abuse )
{
   int aux = 1;
   if ( setsockopt( mSocket, SOL_TCP, TCP_REPAIR, &aux, sizeof( aux )) < 0 )
      reportError( "Tried to do a rude socket close... but could not turn on repair mode.\n" );
}
close( mSocket );

I'll report back if this works. (@edit: tested answer below)

@ The "leave the socket open" idea:

This works but is sub optimal. The attacker has the ability to saturate your system with open sockets. Each request creates a new socket that is left open. With DOS attack you eventually run out of sockets.

Then there is also the problem with managing the open sockets:

1. Just don't close it. Open sockets last forever. Cost for attacker: high - they get no fin. Cost for me: higher. All my file descriptors eventually get used.
2. Spawn a thread per socket to sleep 10 minutes and then close the socket. Cost for attacker: high - they get no fin. Cost for me: higher. While I eventually do close the socket, for each request I have a socket used up for longer than the attacker does, and I have the overhead of a thread.
3. Spawn a thread that handles expiring all abused sockets. Cost for attacker: high - they get no fin. Cost for me: higher. Like 2, lots of sockets held open. Overhead of a single thread to manage it. Code complexity, annoyance.

Answer 1

Ok, did some research and I have an answer that works for me, based on TCP_REPAIR. It is a little more complex than I thought at first:

if ( abuse )
{
   // read some bytes from the spammer - to establish the connection
   u32 tries = 20;
   while ( tries )
   {
      sleep( 1000 );
      char tmpBuf[32];
      s32 readCount = recv( mSocket, &tmpBuf[0], 32, 0 );
      if ( readCount > -1 ) break;
      tries--;
   }
#ifdef TCP_REPAIR
   int aux = 1;
   if ( setsockopt( mSocket, SOL_TCP, TCP_REPAIR, &aux, sizeof( aux )) < 0 )
   {
     reportError( "could not turn on repair mode" );
   }
#else // !TCP_REPAIR
   // no TCP REPAIR - best we can do is an abort close
   struct linger so_linger;
   so_linger.l_onoff = 1;
   so_linger.l_linger = 0;
   if ( setsockopt( mSocket, SOL_SOCKET, SO_LINGER, &so_linger, sizeof so_linger ) < 0  )
   {
      reportError( "Cannot turn off SO_LINGER" );
   }
#endif // TCP_REPAIR
}
close( mSocket );

At the kernel level, the TCP stack will either send a FIN or RST packet if you close a connection, no matter how you do it ( with either close or shutdown ). Either way the attacker is notified that you closed the connection.

We want to silently close the connection and let them wait around to realize you're not answering... because we're vindictive.

TCP_REPAIR is a new socket API designed to allow you to 'freeze' a socket, save its state, and reload the socket state on another process or even another system. Under normal usage the client would never know their connection was transferred elsewhere.

But we can abuse this API, we put the socket in repair mode, but don't save its state and never restore it. When we close the socket in repair mode - it gets silently deleted.

This works for abusive requests that have already begun. That is we've read the spammer's request and decided it was fraud and TCP_REPAIR killed it.

But if you block requests by IP, right after connect, without first reading the socket, somehow the remote party is notified. They get an RST. Or probably something in the connection never quite completes and the remote system aborts the request almost immediately.

So we first read a few bytes from the hacker's socket. In my case the socket is already in non-blocking mode. But if not you want to set the socket to non-block, or else you open yourself up to the hacker opening connection, but sending no packets and leaving your server hanging - like you plan to do to him. If after a few microseconds you don't get a packet you shut him down anyway.

But if you read a few bytes from him, then his program is left waiting for a response from you that never comes.

TCP_REPAIR is only available on Linux Kernels 3.5 and above. Below that the best I can do is a 'dirty' socket close. This is where instead of sending him a FIN you send him and RST. It will look to him like a valid connection was never established. To do this, you turn off SO_LINGER, to essentially break the socket connection close handshake, and then call close.

Works like a charm, point your browser here:

http://oroboro.com/fail

Chrome at least will hang there for 5-10 seconds. Looking at my logs where I was getting 10 hits per second - he's only able to hit me every 10 seconds or so. Load on my system from this: 0.

See ya sucker!

Answer 2

When you detected a malicious client, I would recommend you to not just close the connection, but to also refuse any new connections originating from the same IP address.

What you can do at least is blacklist the IP in your application. Keep a list of banned IP addresses and immediately close any accepted socket which originates from an IP on that list.

But to protect more of your resources, it would be even better to block the connection further outward in the network architecture. When you are able to do so, notify the gateway router to block it. When that's impossible, try to get the load balancer to block it. Failing that, at least add a rule to the local firewall of the server.

But keep in mind that many such attacks originate from consumer-grade internet connections (with or without the user being aware). That usually means their IP addresses are assigned and regularly re-assigned dynamically. An IP which was used by a spammer a few days ago might now be used by a legitimate user. So IP-based bans shouldn't last forever.

Answer 3

Use this:

#include <sys/socket.h>

int shutdown(int socket, int how);

It will send a RST and close (the connection) immediately. This will make it seem, that there's no service on that port and the attacker will hopefully cease spamming on that port. Call close() to free the handle.

Answer 4

When you detect an abuse

have a pool of sockets for "spammers" whenever you detect abuse you put them in the pot using one of those sockets. if no free socket is available recycle the oldest and just shutdown and close it.

if a connection qualifies as !abuse let them use a proper socket.