4

Investigating a slow VPN connection (Cisco ASA IPSec) to a remote office, I noticed on our firewall a lot of access rule matches:

Denied ICMP type=3, code=4 from *ip_address* on interface outside

I noticed that a traceroute to the remote site included the same IP address, somewhere between our ISP and the ISP the remote site uses.

I'm also seeing a message immediately after before saying

No matching connection for ICMP error mesage: icmp src outside *ip_address* dst identity:*firewall_outside_ip_address* (type 3, code 4) on outside interface.  Original IP payload: protocol 50 src  *firewall_outside_ip_address* dst *remote_site_ip_address*

Cisco suggest that this may be symptoms of an attack, but I don't think so.

Protocol 50 is ESP, which is part of IPSec. The remote site is connected to HQ via IPSec VPN using Cisco ASA 5505 at the remote end and ASA 5510 at HQ.

ICMP type=3, code=4 means Fragmentation Needed and Don't Fragment was Set.

Setting Don't Fragment is normal for IPSec ESP packets.

I think what is happening is that packets are leaving our ASA 5510 with the default MTU of 1500. When it hits the router with ip_address that router is unable to pass the traffic to the next hop that uses a smaller MTU, thus requiring fragmentation. The router is sending an ICMP packet back as DF is set, but our Firewall is blocking this, not because of an access rule, but because for some reason our ASA 5510 wasn't expecting this ICMP message.

I am trying to figure out whether the problem is with the configuration on our HQ ASA 5510 (although we have another 36 sites all working fine), the remote ASA 5505 (which is configured uniformly with our other remote ASA 5505s) or something in between the two.

What should I do next?

Update As requested here are the ICMP lines from the HQ ASA 5510:

icmp unreachable rate-limit 1 burst-size 1
icmp permit any echo-reply outside
icmp permit any time-exceeded outside
icmp permit any unreachable outside
dunxd
  • 9,632
  • 22
  • 81
  • 118
  • 2
    IMHO, denying any kind of ICMP packet is just security through obfuscation.. There might have been a point when a ICMP flood could terminate your 28k modem connection, but not in these days.. – pauska Apr 19 '12 at 18:19
  • Absolutely - I am fed up of having to try and convince people of this who were at my level during the "ping of death" days and think ICMP = evil! – dunxd Apr 20 '12 at 09:16

3 Answers3

3

Try setting crypto ipsec df-bit clear-df outside. This won't fix the direct issue here, but may work around it.

As far as the immediate issue - it seems like the ASA isn't realizing that the ICMP packet needs to be used as Path MTU discovery for its tunnel. Check if there's anything in the PMTUD counters displayed by show crypto ipsec sa?

Shane Madden
  • 114,520
  • 13
  • 181
  • 251
  • I'm not seeing PMTUD in the output of `show crypto ipsec sa peer *remote_ip_address*. I am seeing `#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0` and `path mtu 1500, ipsec overhead 74, media mtu 1500`. MTU is set at 1500 (default setting) on ASAs at both ends. – dunxd Apr 20 '12 at 09:39
  • Before I set `crypto ipsec df-bit clear-df outside` can you tell me whether this is going to have a detrimental affect on the majority of IPSec tunnels that aren't having a problem? As far as I can tell this is a global setting. – dunxd Apr 20 '12 at 09:42
  • Yeah, that's the counter you're looking for - 0 received PMTUs means that no discovery is occurring. That configuration only cause problems for other tunnels if they're successfully processing PMTU packets. Check the counters on the existing tunnels. – Shane Madden Apr 20 '12 at 15:11
  • I am seeing the same PMTUs sent/received ais 0 for all SAs. If I run `sh crypto ipsec stats` I can see `PMTUs sent: 149154` and `PMTUs rcvd: 297`. I am seeing some pre-fragmentation but I think this is only happening when packets arrive that are larger than the default MTU of 1500. I guess that means I am safe to switch on the clear-df bit, but have I now discovered an additional issue of PMTU not working correctly on any of my tunnels? Is that something I can resolve, or am I having bad luck with the paths my tunnels are taking? – dunxd Apr 23 '12 at 12:44
  • Ok - have applied `crypto ipsec df-bit clear-df outside` and the messages about that IP address stopped immediately. Happily, all the other tunnels remain up, and appear to be working fine too. So my only remaining concern is that I am now running a sub-optimal configuration, and there may be a better way to resolve the issue with PMTUs not working correctly. Am I now more or less robust than I was with df-bit copied? – dunxd Apr 23 '12 at 13:10
  • Ok - so Cisco say (http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a008081e621.shtml#task2 - see Method 3) "When you encapsulate tunnel mode IPSec traffic, use the `clear-df` setting for the DF bit. This setting lets the device send packets larger than the available MTU size. Also this setting is appropriate if you do not know the available MTU size." Would I be right in reading this as recommendation to *always* clear the DF bit when using IPSEC with Cisco ASA? Or are they saying only in the circumstances detailed (which aren't quite what I am seeing)? – dunxd Apr 23 '12 at 13:45
  • Generally speaking, tunnels usually work just fine with the DF bit copied - but in this case, there's an issue with the PMTUD functioning incorrectly, so it's appropriate to clear it. It almost seems like the ICMP message is being sent directly to the outside interface of the ASA instead of being sent back through the tunnel? So, there might be some more digging to do there to find the root cause, but allowing fragmentation shouldn't hurt anything in the mean time, as long as there's not too terribly much packet loss occurring after the fragmenting. – Shane Madden Apr 23 '12 at 16:59
  • Note that I had some further input from an engineer at Cisco - detailed in my own answer below. I have left @shane-madden's answer as accepted as it was the one that helped me further diagnose the question and go to Cisco with a more accurate description of the problem. – dunxd Apr 26 '12 at 11:35
2

ICMP type 3 code 4 messages are "fragmentation needed but don't fragment set". This means your device sent a packet larger than the MTU of the device sending the ICMP message to you. Normally, the packet could be fragmented, but the DF bit was set. Since you're denying the inbound ICMP message, the ASA doesn't get notified that its packet wasn't delivered. Dropping these ICMP messages is generally bad for performance because it essentially results in packet loss.

Cisco's ASA configuration guide recommends always permitting ICMP type 3 messages, and it specifically mentions that problems can arise with IPsec if these messages are blocked. You can configure the ASA reporting this error to allow them with the following command:

icmp permit any unreachable outside

This only affects ICMP unreachables destined for the ASA itself. If you also need to permit them through the ASA to internal hosts, you'll need to do so with an access-list on your outside interface.

James Sneeringer
  • 6,835
  • 24
  • 27
  • Annoyingly, that line *is* in the ASA 5510 config, and the destination is to the ASA 5510. So I think the denial is explained by the second log entry above. The question is - what should I do next? +1 for supplying the rule the absence of which would normally cause this. – dunxd Apr 19 '12 at 15:24
  • Could you update your question with a sanitized version of the `icmp` statements in the ASA reporting these errors? – James Sneeringer Apr 19 '12 at 17:08
  • Done - see question. – dunxd Apr 20 '12 at 09:26
  • What version is the 5510 running? Cisco bug id CSCsk68658 matches what you're seeing, but it's pretty old, so I don't know if it's what you're actually running into. – James Sneeringer Apr 20 '12 at 14:20
  • Running ASA 8.3(2) so not the latest (8.4), but that bug was apparently resolved with 8.0. So this probably isn't the result of a bug? – dunxd Apr 23 '12 at 12:26
  • No, I wouldn't think so. I'm running 8.3(2) on a pair of 5540's, and I'm not seeing this. I think Shane has you going down the path you need to go. – James Sneeringer Apr 23 '12 at 13:20
1

I had some input from a Cisco engineer who said I had the following options:

  • Setting clear-df as suggested by Shane Madden
  • Setting the MTU on our servers to 1300, so the packets they send are less likely to need fragmentation in the first place
  • Setting the Maximum Segment Size for TCP on the ASA using sysopt connection tcp-mss 1300

I have applied the tcp-mss change, and removed the clear-df command on our HQ ASA and the site with the problems is able to work ok.

This may be a better solution than clearing the df-bit, as that will lead to fragmentation which is not desirable. It is equivalent to setting the MTU on the ASA to 1340 (1300 + 20 bytes for the IP header and 20 bytes for the TCP header) but only affects TCP traffic. It also allows PMTUD to work, which isn't the case when clearing the Don't Fragment bit.

Cisco discuss this option in detail in Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC.

dunxd
  • 9,632
  • 22
  • 81
  • 118
  • Related to Cisco's "suggestions" and why they a bad idea (which you probably already knew): [What exactly is the MTU/MRU issue, what it is caused by and how to fix it?](http://serverfault.com/questions/376752) – Chris S Apr 26 '12 at 12:38
  • 1
    I should add that the engineer was just outlining the different things I could do myself to mitigate against the problem of broken PMTUD. He was saying that if I can't do anything about fragmentation on route, that either of last two options would result in the packets not needing to be fragmented after they had been encrypted. The first option would result in large packets getting fragmented on route, which is not good for encrypted packets - see the article linked above. – dunxd Apr 26 '12 at 12:53
  • @ChrisS have read your very good answer to that other question. It doesn't look like anything on route was actually doing anything wrong according to the RFCs, but for some reason yet to be established, my ASA didn't like the ICMP Destination Unreachable packets it was getting - but it certainly wasn't caused by the ASA config specifying not to allow ICMP packets. Any input on what other diagnostics or actions I might take would be very gratefully received. – dunxd Apr 26 '12 at 13:20
  • @dunxd Regarding the root cause question - what device is throwing the fragmentation needed packets at you, anyway? Is it an internet router in between you and that remote site, or something more odd? – Shane Madden Apr 26 '12 at 15:15
  • Internet router in between me and remote site. It's not a router owned by either of the ISPs from which we buy services. It is intermediate (I'm assuming a backbone provider). – dunxd Apr 27 '12 at 10:31