Find whats using all my servers bandwidth

Question

TLDR: how can I dig deeper into my Mac mini OSX server to determine what processes are consuming so much bandwidth, or where all the inbound traffic is coming from?

Resource caching is on, all resources are minified or images crushed, our pages are less bandwidth consuming then the website last month (www.vulytrampolines.com), and our traffic in analytics is the same.

Back story: We have 2x mac mini servers running our website/staging/databases etc. Since moving from a dedicated server in America to a colocation place in the city, our bandwidth consumption has somehow quadrupled. There is a database replication process setup between the two, as well as DNS and various websites (e.g. large files, database tasks, intranet packages etc are on staging server, website and database are on production server)

Our staging server has had 7GB of inbound traffic in 3 DAYS. Does anyone know how to check for inbound traffic sources to see potentially where this consistent stream of inbound traffic at a 200k connection is coming from? We have no idea. We aren't sending it files at all, the only thing that should be active is SSH and a database replication process. netstat [see below] shows we have about 20+ established and 30+ close_weight connections to port 625. We haven't the foggiest how this could be happening.

The annoying thing is, webstats show we haven't used anywhere near 11.66GB for HTTP traffic (it says we used 22GB in the last month, yet our outbound traffic was recorded as over 100GB). Database stats say we haven't used anywhere near enough bandwidth to cause the issue either.

This is our staging server, venus1 (it has been like this for a few weeks):

This is our production server, venus2:

netstat -anp tcp output on venus1. Most of the foreign addresses are from our work IP addresses. 11211 is memcached.

tcp4       0      0  122.99.117.18.49712    204.93.223.143.80      ESTABLISHED
tcp4       0      0  122.99.117.18.11211    122.99.117.18.49711    ESTABLISHED
tcp4       0      0  122.99.117.18.49711    122.99.117.18.11211    ESTABLISHED
tcp4       0     52  122.99.117.18.22       59.167.152.67.56106    ESTABLISHED
tcp4       0      0  122.99.117.18.11211    122.99.117.18.49588    ESTABLISHED
tcp4       0      0  122.99.117.18.49588    122.99.117.18.11211    ESTABLISHED
tcp4       0      0  122.99.117.18.49410    122.99.117.19.3306     ESTABLISHED
tcp4       0      0  122.99.117.18.5432     122.99.117.18.58704    ESTABLISHED
tcp4       0      0  122.99.117.18.58704    122.99.117.18.5432     ESTABLISHED
tcp4       0      0  127.0.0.1.58699        *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      110.142.234.238.62055  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.71.206.61838    ESTABLISHED
tcp4       5      0  122.99.117.18.625      203.206.171.34.61741   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.61270  ESTABLISHED
tcp4       0      0  127.0.0.1.54           *.*                    LISTEN
tcp4       0      0  122.99.117.18.53       *.*                    LISTEN
tcp4       0      0  127.0.0.1.53           *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      110.142.234.238.63980  ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.55282   ESTABLISHED
tcp46      0      0  *.80                   *.*                    LISTEN
tcp4       0      0  *.*                    *.*                    CLOSED
tcp46      0      0  *.443                  *.*                    LISTEN
tcp4       0      0  *.*                    *.*                    CLOSED
tcp4       0      0  122.99.117.18.625      122.99.117.19.50766    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.63981    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.60214  ESTABLISHED
tcp4       0      0  122.99.117.18.625      27.33.32.204.65196     ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.60274  ESTABLISHED
tcp4       0      0  122.99.117.18.625      122.99.117.19.53201    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.59662   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.49869    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.53827    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.64678  ESTABLISHED
tcp4       0      0  122.99.117.18.625      122.99.117.19.52810    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.62510  ESTABLISHED
tcp4       0      0  122.99.117.18.625      122.99.117.19.49909    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.56096  ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.53247   ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.62051   ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.93.92.59123     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.57173    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.49352   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.64362  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.59772     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.59528    CLOSE_WAIT
tcp4       0      0  *.3306                 *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      27.33.32.204.56812     ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.52987  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.50598    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.63339  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.63283  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.61312    ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.52045  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.49172    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.50501   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.93.92.56042     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.55882     ESTABLISHED
tcp4       0      0  122.99.117.18.311      58.111.93.92.55875     ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.58776   ESTABLISHED
tcp6       0      0  *.5432                 *.*                    LISTEN
tcp4       0      0  *.5432                 *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      58.111.93.92.52692     ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.57121   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.54673     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.53915     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.52109     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.51807     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.65049     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.93.92.64442     ESTABLISHED
tcp4       0      0  122.99.117.18.311      203.206.171.34.51628   ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.51594   ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.62597     ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.62454     ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.58088     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.57305     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.53724     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.62224     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.62064     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.58236     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.51320   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.51297   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.50864   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.49800  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.50894     ESTABLISHED
tcp4       0      0  122.99.117.18.625      59.167.152.67.50411    ESTABLISHED
tcp4       0      0  122.99.117.18.625      27.33.32.204.54446     ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.51680     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.60797     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.60729     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.49209    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.63371     CLOSE_WAIT
tcp4      81      0  122.99.117.18.625      113.128.44.66.3842     CLOSE_WAIT
tcp4      11      0  122.99.117.18.625      213.57.189.27.55646    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.53655  ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.53644  ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.52146     CLOSE_WAIT
tcp4       0      0  127.0.0.1.8005         *.*                    LISTEN
tcp46      0      0  *.8009                 *.*                    LISTEN
tcp46      0      0  *.8080                 *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      58.111.79.42.50716     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.49872    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.63218     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.62471     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.64758     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.64646     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56788     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56770     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56017     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.53131     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.52519     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.51215     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.51131     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.57058    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.56711    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.50975     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.57209     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.54753     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.60786     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56174     CLOSE_WAIT
tcp4       0      0  *.11212                *.*                    LISTEN
tcp6       0      0  *.11212                *.*                    LISTEN
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49167        ESTABLISHED
tcp4       0      0  127.0.0.1.49167        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  122.99.117.18.5218     122.99.117.18.49166    ESTABLISHED
tcp4       0      0  122.99.117.18.49166    122.99.117.18.5218     ESTABLISHED
tcp46      0      0  *.5268                 *.*                    LISTEN
tcp46      0      0  *.5218                 *.*                    LISTEN
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49163        ESTABLISHED
tcp4       0      0  127.0.0.1.49163        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49162        ESTABLISHED
tcp4       0      0  127.0.0.1.49162        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49161        ESTABLISHED
tcp4       0      0  127.0.0.1.49161        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  127.0.0.1.5348         *.*                    LISTEN
tcp4       0      0  *.11211                *.*                    LISTEN
tcp6       0      0  *.11211                *.*                    LISTEN
tcp4       0      0  *.88                   *.*                    LISTEN
tcp6       0      0  *.88                   *.*                    LISTEN
tcp6       0      0  *.2000                 *.*                    LISTEN
tcp4       0      0  *.2000                 *.*                    LISTEN
tcp6       0      0  *.4190                 *.*                    LISTEN
tcp4       0      0  *.4190                 *.*                    LISTEN
tcp4       0      0  *.464                  *.*                    LISTEN
tcp6       0      0  *.464                  *.*                    LISTEN
tcp6       0      0  *.25                   *.*                    LISTEN
tcp4       0      0  *.25                   *.*                    LISTEN
tcp4       0      0  *.749                  *.*                    LISTEN
tcp6       0      0  *.749                  *.*                    LISTEN
tcp4       0      0  *.22                   *.*                    LISTEN
tcp6       0      0  *.22                   *.*                    LISTEN
tcp4       0      0  *.5900                 *.*                    LISTEN
tcp6       0      0  *.5900                 *.*                    LISTEN
tcp4       0      0  *.625                  *.*                    LISTEN
tcp6       0      0  *.625                  *.*                    LISTEN
tcp4       0      0  127.0.0.1.631          *.*                    LISTEN
tcp6       0      0  ::1.631                *.*                    LISTEN
tcp4       0      0  *.311                  *.*                    LISTEN

Answer 1

Unless you have a very good reason not to, please firewall all non-public ports on your Mac Minis immediately and only expose what you need to! I did a quick check and all of your services currently appear to be completely open to the internet (SSH, SMTP, Tomcat, VNC, MySQL and apple-xsrvr-admin (TCP625) certainly are), which is a very bad idea.

The large amount of inbound traffic you're seeing may quite possibly be hackers/bots trying to brute-force logins and passwords for your services – I see this on servers a lot, particularly against SSH (as it's often public-facing) and directly against popular web software, such as WordPress's /wp-login.php page. From your netstat I can see that IPs from Israel and China were trying to access TCP625 (apple-xsrvr-admin, used for DirectoryService, Open Directory Assistant & Workgroup Manager), which isn't a good sign. I hope all of your usernames and passwords are strong, because – not trying to sound alarmist here, but – 7GB's worth of brute-forcing may have let some bad people get to things they shouldn't have access to, regardless of the security/patch state of any software.

Check with your hosting provider/colo that you have some sort of remote serial console and/or VNC access should you accidentally block yourself from accessing your servers, then add the absolute minimum required for you to remote admin the servers yourself to the rulesets and turn the firewalls on, which I'm assuming they're not already. By default, OS X Server's firewall blocks all incoming ports except those used to configure the server remotely (TCP22, TCP311, TCP626, TCP625, ICMP standard ping (in & out), UDP53 DNS name resolution) so you should be OK to turn it straight on, though you want to lock these down more once you've done so. Create Address Groups that are specific to your office IP (or your ISP's IP netblock(s) if your office doesn't have a static IP address) and use these to open admin access (e.g. SSH, OpenDirectory, VNC) to these only. Create another Group with single IP access for 122.99.117.18 and 122.99.117.19 (or 122.99.117.18/31 mask) to talk to each other and allow this for the MySQL replication. Open the SMTP, HTTP, HTTPS ports to the world, assuming they're public-facing. Lock everything down tight, granting permission only to the IPs that need access to each port. Consider doing this for outbound traffic, too. You want to plan this in advance and make sure you get it all right, but do it soon rather than leaving the server an open sitting duck.

Review your servers' logs and look for suspicious activity. In particular look for any successful logins to services from odd locations or at odd times. Establish some procedures to do this regularly.

I'm not sure which version of OS X Server you have, but guessing at 10.6 or 10.7, this Peachpit document may help get you started on the firewalling front.

As may this Apple support PDF on Networking Services (see chapter 4 for firewalling).

For more advanced firewall configuration, try Waterroof or Icefloor, which provide a simple GUI rather than having to mess with pf on the commandline.

---

(Edit to address the ipfilter rules pasted from each server)

OK, let's start with venus1 (122.99.117.18). The obvious problem here is that there's no catch-all deny ip from any to any, so we need to fix that immediately. From the commandline, you can issue: sudo ipfw add 65534 deny log logamount 1000 ip from any to any

Or do it from the Server Admin's Settings tab, under the Advanced sub-tab if you're not too comfortable with manipulating ipfw rules from the commandline (which should make you nervous as it can end badly if you slip up). There should be a list of Advanced Rules somewhat similar to the following image. Assuming that rule is there already at the bottom, tick it to enable it:

Remember that the ipfw rules flow down the priority list, so a priority 1 rule takes precedence over everything else. So adding a 'deny everything' rule down at 65534 is last on the list, meaning it'll only deny a connection if everything above it doesn't allow something first.

OK, with that out of the way, let's deal with some Address Groups. You've defined an Address Group that's 122.99.117.16/29, so 122.99.177.16-22 (where one IP's the gateway), which is your Mac Minis and other co-lo IP addresses. You probably want to define another Address Group for your office location, which I'm guessing at a static IP of 59.167.152.67 (or if your work gets a dynamic IP then the whole netblock of Internode is 59.167.0.0/16). Do this for both venus1 & venus2 and add more for any other offices you have.

Once you've definied your office Groups then give each of these permission to access TCP22, TCP311, TCP626, TCP625, TCP3283 and TCP5900, so you can always admin your Minis even when you disable world-access to these ports. Using the GUI, go to the Services sub-tab of the Server Admin's Settings. Change the "Editing Services for:" to your office Address Group(s), then tick the relevant boxes for the above ports.

Now let's address some more problematic venus1 rules. For instance:

12321    1570      78548 allow tcp from any to any dst-port 3306

Rule 12321 allows everything to connect to MySQL. Disable this rule for the "any" Address Group (using the Services sub-tab again) and while you're there, any other services that the world doesn't need access to. You probably want to keep TCP25 (smtp mail), TCP/UDP53 (dns), TCP80 (HTTP), TCP443 (HTTPS) world-accessible, but do you need world-access to TCP8080 (tomcat's server status)? Anything else? Lock it all down and only grant access to your office Address Group(s) and co-lo where appropriate (e.g. the MySQL replication).

Remember to test everything! Learn to use nmap to scan your server ports so you can see what's open and closed. Scan from different locations (e.g. your office, ssh'ed into your other server, at home through another ISP, etc.) to make sure the rules are working as they should. Be careful not to lock yourself out, but do lock everything down as tight as you can. Re-read the Peachpit doc and Apple support PDF linked above and really learn the material, it'll stand you in good stead. Oh, and keep a regular eye on your logs!

OK, hopefully that should see you right, or at least put you on a better path. Best of luck.

Answer 2

The quickest solution would just to use a packet sniffer - tcpdump or wireshark (the latter having a graphical interface). If there's some kind of constant traffic source it will tend to stick out. You'll also be able to dig into the particulars of the payload once you do discover the source of the unknown traffic.

Answer 3

Based on the close correlation between the traffic stats on the staging and production servers, it appears your two servers are talking to each other.

Port 625 is a management port used by Apple products. From your netstat, it appears that one machine (.19) is connecting to the other (.18) on port 625. From some Google searches, it appears that Mac OS X uses port 625 for Workgroup Manager (which was replaced in Mountain Lion with Profile Manager).

It also appears that a wide variety of other machines on the Internet are connecting to port 625 as well, probably as break-in attempts. You should firewall your server as soon as possible to prevent intrusion.