Traffic backlogged after (D)DoS attack?

Question

About 25 hours ago, I got an e-mail from UptimeRobot (uptimerobot.com) telling me that me websites had gone down.

I raced over to my server room, where my TP-Link router was completely unresponsive (even from the LAN side), traffic lights on the router where blinking at an ungodly speed, etc. I got a message from a "friend" saying that somebody he knew was an idiot and had gone on some chat channel and told everybody to attack me.

After 17 hours of getting hammered with traffic (around 15,000 packets per second), I moved my network to a new WAN IP, which obviously stopped the attack. To the best of my knowledge, those unpleasant people are still attacking the old IP. I feel bad for whoever that address gets allocated to.

Anyways, after I switched IPs, the traffic lights were still blinking on my router, and it was still unresponsive. I gave it and the modem both a hard reboot; and after they came back up the traffic lights had slowed to a normal speed and the router is now responsive.

The problem is, the network still feels like it's under attack - whenever I ping google.com, (which is obviously up), I get about 50% packet loss and the round trip takes around 100ms - compared to the normal 30ms or so.

I've rebooted the router and modem several times, thinking that maybe they were still 'clogged up' with traffic from the attack; but this has proved unsuccessful. The LAN itself is fine - I get 0% packet loss between different nodes with latencies as low as 0.051ms, so I've deduced that nothing within the internal network is causing the problem.

Does anybody have any idea as to what's going on here? Is it possible that the pipe to our ISP is completely backlogged with traffic sent during the attack? None of this makes sense to me, just wondering if anybody has experienced anything like this before. Thanks in advance.

EDIT:

Here is some output after pinging the network from the outside, in case anyone's interested:

(IP omitted because I'm paranoid now after that attack)

PING x.x.x.x (x.x.x.x): 56 data bytes
64 bytes from x.x.x.x: icmp_seq=0 ttl=59 time=47.797 ms
64 bytes from x.x.x.x: icmp_seq=1 ttl=59 time=47.103 ms
64 bytes from x.x.x.x: icmp_seq=2 ttl=59 time=41.792 ms
64 bytes from x.x.x.x: icmp_seq=3 ttl=59 time=51.739 ms
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
Request timeout for icmp_seq 6
Request timeout for icmp_seq 7
64 bytes from x.x.x.x: icmp_seq=8 ttl=59 time=43.218 ms
Request timeout for icmp_seq 9
64 bytes from x.x.x.x: icmp_seq=10 ttl=59 time=45.034 ms
64 bytes from x.x.x.x: icmp_seq=11 ttl=59 time=42.263 ms
Request timeout for icmp_seq 12
64 bytes from x.x.x.x: icmp_seq=13 ttl=59 time=46.498 ms
Request timeout for icmp_seq 14
Request timeout for icmp_seq 15
Request timeout for icmp_seq 16
64 bytes from x.x.x.x: icmp_seq=17 ttl=59 time=43.183 ms
^C
--- x.x.x.x ping statistics ---
18 packets transmitted, 9 packets received, 50.0% packet loss
round-trip min/avg/max/stddev = 41.792/45.403/51.739/3.031 ms

This is from a location that is geographically close to the network, and normally has 0% packet loss with latencies around 25ms. Thanks again in advance for your help.

Here is a traceroute, it's probably more informative than the ping output above:

traceroute to x.x.x.x (x.x.x.x), 64 hops max, 52 byte packets
 1  10.0.0.1 (10.0.0.1)  4.767 ms  4.178 ms  4.195 ms
 2  7.38.4.1 (7.38.4.1)  11.357 ms  18.649 ms  14.658 ms
 3  69.63.243.209 (69.63.243.209)  15.616 ms  32.639 ms  16.381 ms
 4  69.63.249.85 (69.63.249.85)  26.902 ms  13.912 ms  15.729 ms
 5  67.231.220.182 (67.231.220.182)  26.328 ms  27.733 ms  15.328 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
^C

It looks like the packets are being dropped after 67.231.220.182, which according to ARIN's WHOIS is a Rogers switching station on Mt. Pleasant - which should be the last hop between where I am and the network I'm trying to reach. This leads me to believe that it's not my ISP's problem, so I don't think my pipe is the problem.

Are you really providing an IP-only service without any domain name relation? If 'switching to another WAN IP' also means you switched DNS records, the case is quite clear, no? — Karma Fusebox, May 16 '13 at 00:31
@KarmaFusebox - I am providing a service with domain names, but did not update the DNS records of the domain which was getting attacked - either way, I know they were attacking my IP directly; because during the attack I changed my DNS records to point to a Google IP, which did not stop the attack. Either way they have no way of knowing the new IP. — Libbux, May 16 '13 at 01:04
Well then. The ping from outside you gave is not much of use. Do a traceroute. Maybe the timing is just funny enough that your ISP has a totally unrelated problem right now? In any case, traffic isn't backlogged or queued in such dimensions. The traceroute should show you where the packets get lost. — Karma Fusebox, May 16 '13 at 01:21
(And maybe you should clarify a bit if this is indeed a professional scenario (TP-Link/Modem?) as the folks might kick you over to SU or just close your question as off topic.) — Karma Fusebox, May 16 '13 at 01:24
@KarmaFusebox - This is a small-ish business - we don't have a fibre connection, we have a coaxial cable connection coming to our demarc point. We have a TP-LINK TL480T+ load-balance router acting as our primary router, firewall, DHCP server, etc. I'm beginning to believe that the router may still be flooded or overloaded, even after the reboots. — Libbux, May 16 '13 at 01:32
All right, then hardware's quite a hint to look further into. Unfortunately I'm not familiar with those models or their software internals. Hope someone else around here is! — Karma Fusebox, May 16 '13 at 02:09
@KarmaFusebox Ok, thanks for the help. Do you think the hardware could have actually been damaged? I don't think it overheated or anything, but do you think it could be damaged from being overloaded for too long? — Libbux, May 16 '13 at 03:13
"o the best of my knowledge, those unpleasant people are still attacking the old IP. I feel bad for whoever that address gets allocated to." You said that so blithely! I need to get some needle and thread to stitch the ass I just laughed clean off and this comment will prolly get deleted by then, but thank you sincerely for making my evening with that one and stars / upvotes from me abound as a result ! — Karen3819x4, May 16 '13 at 04:20

score 4 · Accepted Answer · answered May 16 '13 at 04:14

No, routers and modems don't get "queued up", they only pass traffic. If they've been rebooted, they start fresh and listen for incoming traffic. If they get a packet, they route it, and if there's nothing on the other end, it just flows in to nothing and the router or modem doesn't cache a copy of the packet for retrying or anything like that. Any network retries, caching, tracking lost packets, etc, is the responsibility of the endpoints, not the equipment in between.

As for solving your packet loss problem, this is really something you should be calling your ISP about. It's their job to ensure you have a reliable and stable connection. And for business-class customers they're usually very helpful and would probably even engage rate limiting, IP blocking or some other measure to stave off the attack. Get well acquainted with your ISP's helpdesk. Remember, you're paying them to manage your network connectivity, put them to work! :)

Traffic backlogged after (D)DoS attack?

1 Answers1