Cisco ASA: Terrible network speed once or twice a day, fine otherwise

Question

I've been trying to get to the bottom of this for a while now and, well, it seems like I can't. We have two servers behind an ASA5505 (software version 8.3) at a datacenter. They run a wide variety of services, including our website, internal XMPP server, game servers (Minecraft and Team Fortress 2, both using UDP for the most part), mail...

Every day roughly around noon PST the network speed becomes absolutely terrible for about an hour while the system load of the firewall goes from the usual 30% to above 80%. According to show processes cpu-hog, "Quack process" (what the duck?!) and especially "Dispatch Unit" are, well, hogging the CPU a bit.

There seems to be a pattern when the network goes bad. For about 2 seconds it's at full speed, then it slows down to almost a halt for 2 more. I've enabled logging to ssh during this, and nothing interesting showed up. Just a few blocked ICMP requests and, a bit odd, Deny IP due to Land Attack from [one of our IPs] to [the exact same IP], but that might be an actual attack?

Anyways, speed is bad from and to the two servers and also to the firewall itself, making me thing that it's overburdened, although the ping between the two servers is always good. I'm not sure how the network is set up exactly though, so there might simply be a small switch between the firewall and the servers.

Another weird thing, but, again, this might be normal (couldn't find anything about it), in show threat-detection statistics the internal IPs of our servers/VMs show up first and some actually have numbers greater than 0 for fw-drop.

What should I try next time this issue comes around? Any ideas as to what might cause this? Should I disable limit-policy-map (see below)?

EDIT: Pinging the servers from the firewall will also show these symptoms.

Here's some more system information:

access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
            alert-interval 300
access-list outside_in; 33 elements; name hash: 0xc5896c24
access-list outside_in line 1 extended permit tcp any object-group www_servers object-group www_srv 0x9c6770f3 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq ftp (hitcnt=2443) 0x73b87a74 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq ssh (hitcnt=27915) 0x73a19ab3 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq www (hitcnt=21568957) 0x045edf43 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq https (hitcnt=19746) 0xe54a2315 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 3389 (hitcnt=3919) 0x58629d3c 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 30 (hitcnt=134) 0xcd3db679 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 5922 (hitcnt=43) 0x17c6f16b 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 6122 (hitcnt=1) 0x3ea3c2e6 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 2200 (hitcnt=2) 0x8356fbc6 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 5722 (hitcnt=1) 0xaefada3e 
  access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq domain (hitcnt=17) 0x45c7e0b1 
access-list outside_in line 2 extended permit udp any object-group www_servers object-group www_srv_udp 0x9426d24f 
  access-list outside_in line 2 extended permit udp any(65536) object-group www_servers(1) eq 3389 (hitcnt=1) 0x15cdc545 
  access-list outside_in line 2 extended permit udp any(65536) object-group www_servers(1) eq domain (hitcnt=4468079) 0x1b6d6b19 
access-list outside_in line 3 extended permit icmp host [...] any (hitcnt=0) 0x155d597f 
access-list outside_in line 4 extended permit icmp host [...] any (hitcnt=289) 0x0fcc844a 
access-list outside_in line 5 extended permit icmp any object-group www_servers echo-reply 0x46f79e30 
  access-list outside_in line 5 extended permit icmp any(65536) object-group www_servers(1) echo-reply (hitcnt=97) 0x53984766 
access-list outside_in line 6 extended permit tcp host [...] eq 25565 host 10.5.209.12 eq 25565 (hitcnt=0) 0x60c828e6 
access-list outside_in line 7 extended permit tcp any object-group mc eq 25565 0xcb0d2f17 
  access-list outside_in line 7 extended permit tcp any(65536) object-group mc(6) eq 25565 (hitcnt=478488) 0x3ce89b9a 
access-list outside_in line 8 extended permit tcp any object-group irc object-group ircd 0x65619a8f 
  access-list outside_in line 8 extended permit tcp any(65536) object-group irc(8) eq 6667 (hitcnt=6336) 0xda23eb42 
  access-list outside_in line 8 extended permit tcp any(65536) object-group irc(8) eq 6969 (hitcnt=8445981) 0xb39f9de5 
access-list outside_in line 9 extended permit tcp any object-group rob object-group xmppd 0x24db3318 
  access-list outside_in line 9 extended permit tcp any(65536) object-group rob(9) eq 5222 (hitcnt=2836) 0x3b220aef 
  access-list outside_in line 9 extended permit tcp any(65536) object-group rob(9) eq 5269 (hitcnt=316) 0x8c4a1677 
access-list outside_in line 10 extended permit udp any object-group rob object-group xmppd 0x56997935 
  access-list outside_in line 10 extended permit udp any(65536) object-group rob(9) eq 5222 (hitcnt=0) 0x1378a09e 
  access-list outside_in line 10 extended permit udp any(65536) object-group rob(9) eq 5269 (hitcnt=0) 0x484e999c 
access-list outside_in line 11 extended permit udp any object-group tf2_servers object-group tf2_udp_ports 0x4ed88dd7 
  access-list outside_in line 11 extended permit udp any(65536) object-group tf2_servers(12) range 26901 27009 (hitcnt=20) 0x984f0cfd 
  access-list outside_in line 11 extended permit udp any(65536) object-group tf2_servers(12) range 27015 27024 (hitcnt=1842395) 0x5117dbf3 
access-list outside_in line 12 extended permit tcp any object-group tf2_servers object-group tf2_tcp_ports 0xd792e8d1 
  access-list outside_in line 12 extended permit tcp any(65536) object-group tf2_servers(12) eq 8080 (hitcnt=16028) 0x1f9dcdd6 
access-list outside_in line 13 extended permit object-group tcp_udp any object-group rob object-group mumble_ports 0x62e8f226 
  access-list outside_in line 13 extended permit tcp any(65536) object-group rob(9) eq 64738 (hitcnt=4) 0x663e2204 
  access-list outside_in line 13 extended permit udp any(65536) object-group rob(9) eq 64738 (hitcnt=14) 0x3751c05a 
access-list outside_in line 14 extended permit udp any object-group kfy_servers object-group kfy_ports 0x928ebaab 
  access-list outside_in line 14 extended permit udp any(65536) object-group kfy_servers(16) eq 9009 (hitcnt=52) 0x3c77464e 
access-list outside_in line 15 extended permit udp any host 10.5.209.10 object-group bittorrent 0x20a28a30 
  access-list outside_in line 15 extended permit udp any host 10.5.209.10(168153354) eq 10299 (hitcnt=44693845) 0x140f0e51 
access-list outside_in line 16 extended permit tcp any host 10.5.209.10 object-group bittorrent 0xfe939491 
  access-list outside_in line 16 extended permit tcp any host 10.5.209.10(168153354) eq 10299 (hitcnt=3763575) 0x1ef0e366 
access-list outside_in line 17 extended permit icmp any object-group rob 0x6f990c22 
  access-list outside_in line 17 extended permit icmp any(65536) object-group rob(9) (hitcnt=1418) 0x8401a397 
access-list limiter; 3 elements; name hash: 0x189b5c6d
access-list limiter line 1 extended deny ip host [...] any (hitcnt=0) 0x72cb4f57 
access-list limiter line 2 extended deny ip host 10.0.0.0 any (hitcnt=0) 0x3d376866 
access-list limiter line 3 extended permit ip any any (hitcnt=89047566) 0x1bc67ee2 


policy-map limit-policy-map
 class limit-map
  set connection per-client-max 500 per-client-embryonic-max 30 
  set connection timeout embryonic 0:00:10 half-closed 0:05:00 dcd 
policy-map global_policy
 class inspection_default
  inspect dns 
  inspect ftp 


class-map limit-map
 match access-list limiter
class-map inspection_default
 match default-inspection-traffic
class-map limit

So far so good, no slowdown today. Maybe it was the limit stuff. Which would be kinda sad. — kshade, Jul 12 '13 at 19:31

score 0 · Accepted Answer · answered Jul 12 '13 at 12:46

You realise that the throughput of the ASA5505 is in the 10s of mbits? They were designed for small office/home office and branch offices. They were never designed to handle gigs of traffic.

Anyway, the ASA5505s have a number of factors which could cause increased CPU load. Most of em are filter based. If you've got complicated filters and policies in place. The more complicated things you do in these filters, the more processing time each packet will consume.

I would start by looking at traffic graphs for upstream and the servers to find out of ere are increased levels of traffic at the times you specify. You're looking for patterns really. If you don't have graphing for your servers, you should get some and your providers should be able to give you some form of traffic data. This should give you some indication of in which direction the problems is sourced.

If its on the server side, then you have everything under your control and should look for e culprit there. Perhaps an errant process, or dodgey cron job? Perhaps some process that for some reason is now generating lots of traffic?

If its a provider side problem, then you'll have to consult with them to see if Rees anything that can be done.

Our host rented us this thing for quite a bit of money and refuses to help any further since we've got administrative access instead of having them manage it (for even more fees). It's on a 100 megabit connection, according to Cisco's documentation that should be alright? [link](http://www.cisco.com/en/US/products/ps6120/prod_models_comparison.html). There are traffic graphs, but nothing suspicious really. Same with the servers, they don't generate a lot of traffic or anything while this happens. — kshade, Jul 12 '13 at 14:14
Are the servers connected to the ASA5505 directly or via some intermediary switch? — pobk, Jul 12 '13 at 14:19
I'm actually not sure, but I'm guessing there is a switch if the ASA doesn't have a built-in one that's independent of the actual firewall hardware. — kshade, Jul 12 '13 at 14:21

Cisco ASA: Terrible network speed once or twice a day, fine otherwise

1 Answers1