My Ubuntu 10.04 server kills all WAN bandwidth when it's attached to my LAN. Where do you begin troubleshooting?

Question

First I should say that my Linux knowledge is minimal; just enough to set up some servers (Apache, Tomcat, Couch, etc). I built a MiniITX server to host some simple sites, act as an SSH tunnel while I'm away, and act as a torrent server. It was not properly secured for a long time (iptables was empty, all ports open, no firewall) though my router did not have much port forwarding set up beyond HTTP, FTP, and SSH.

A week or two ago my bandwidth at home dropped from around 27Mbps to 2Mbps and my upload went from 7Mbps to 0.06Mbps. When I unplug the server from the LAN, by bandwidth shoots back up.

I threw up a restrictive iptables, removed most of the port forwarding, and checked my router logs to see if there were any open connections from the server (malware?) but there were none.

What would you do? What are the first things you'd check?

I can of course reinstall everything from scratch, but I'd like to find the root cause.

EDIT

Connected to LAN:

sudo route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     *               255.255.255.0   U     0      0        0 eth0
default         192.168.0.1     0.0.0.0         UG    100    0        0 eth0

iptraf > IP traffic monitor > eth0

which shows me x103 (the problem server) and x130 (the Mac from which I'm SSHing in) with Packets and Bytes just flying up at a constant rate, never stopping. I'm guessing that this is an infinite feedback loop, where any iptraf update needs to be sent over the wire, resulting in another update being logged, etc. Anyway, it's showing a TCP flow rate of 26 kbits/s which simply cannot account for the multi-Mbps drop in both upload and download.

iptraf > Detailed interface statistics > eth0

iptraf > Statistical breakdowns > By TCP/UDP port > eth0

Answer 1

I would unplug it immediateley. Maybe you are acting as a spam relay or something worse. Then check locally.

It would also be better to boot from a live linux (like knoppix or ubuntu live or your preferred brand), do not boot it from the hard disk any more.

It is not easy to diagnose without further knowledge, but most probably someone has taken over this machine and is using it for something. Could also be a botnet or be used for dos attacks.

I would suggest to isolate this machine from everything else. And maybe other machines could also be infected.

Make a backup of your data. Use security software, like virus checkers. Learn how to build a protected environment (machines, lan, firewall, router) and start fresh. Be careful when using your data from these machines, could be infected.

There is a potential risk in such setups that you will be held responsible for forbidden or criminal activity that has been done by someone else using your machine and IP, also your internet service provider does not like what happened (if this really happened).

Do not feel guilty as this happens even to many experienced admins but try to act responsible and also learn from it :) Good luck!

Answer 2

TCPDUMP can show you the traffic on the wire. This can get messy, so you may want to redirect it to a file and browse it later:
tcpdump -s 0 -Ai eth0

-s 0 sets the length of the packet capture, 0 means as much as possible so feel free to adjust as needed.
-A prints the traffic in ASCII so you can read some of it. i eth0 sets the interface to watch. You should make sure this is your WAN interface.

With any luck you'll see what's generating your bandwidth problems.

Answer 3

What is the output of sudo route before and after the LAN is plugged?

EDIT: Eth0 is you're LAN, where is the interface for the WAN - can you post the output of ifconfig -a and route (again) when both networks are operational?

Don't worry about viruses, backdoors, or spam forwarding - so far I don't see a good reason that you have it. Lets just diagnose this as a networking problem.

The iptraf you are showing adds up only to about 60K bits/sec - that's not a lot of traffic. You were talking about a bandwidth dropping from 27M to 2M bits/sec. Please reproduce the problem (plug-in the public interface, etc.) and then run iptraf, ifconfig -a, and route.

Answer 4

tcpdump as others have suggested, although if you suspect that the box may be rooted, I'd want to capture traffic from another trusted machine that's running as a bridge between the box in question and the LAN switch (built-in to the router or otherwise) or if you have an old hub, you can use that too.

You mention that it's a torrent server: are you by chance seeding a pile of torrents? That'll quickly kill an Internet connection if there's too many peers and/or too much bandwidth consumed.

Answer 5

If you're willing to risk running it live again (see mit's answers), you can get an immediate sense for what's going on by running iptraf.

tcpdump will give you a more definite answer (easier to understand in wireshark)

Answer 6

After running the tests given in the various answers, it became clear that my Linux box wasn't actually transmitting much of anything. This left me with the router - a D-Link DIR-655 A3 - as the problem. It's off the shelf (not customized) and it's worked fine for years. I spend some time playing with my Windows Media Center PC which is also connected to the router. I noticed the bandwidth drop happen again - with the server unplugged - once I started a torrent client. I know it isn't my ISP throttling the bandwidth when it notices torrents, because when I connect directly to the modem the problem disappears.

I think the problem is my router's QoS Engine or WISH (Wireless Intelligent Stream Handling) capabilities going crazy. Both of these services shape traffic, both LAN and WAN traffic. I disabled both services and so far the problem seems to have gone away.

I want to thank everyone who posted a suggestion; these tips helped me discover that the problem wasn't actually with my server.