Generating ICMP packets when TTL=2?

Question

By inspecting the payload of ICMP time-exceeded packets, I noticed that sometimes it is the last but one router (when ttl=2 in the returned packet) or even a previous one(up to 5 hops before, ttl=5) that drops the packet and generates an ICMP message.

How so? Any reason behind this?

How do you set this in a CISCO router?

Edit:

please note that ALL these packets are ICMP type 11 code 0, which means:

type = time-exceeded, code = ttl-zero-during-transit

Edit2: Here are two examples of such ICMP packets.

###[ IP ]###
  version   = 4L
  ihl       = 5L
  tos       = 0x0
  len       = 168
  id        = 9969
  flags     = 
  frag      = 0L
  ttl       = 243
  proto     = icmp
  chksum    = 0x19ea
  src       = 193.51.189.25
  dst       = 134.59.129.241
  \options   \
###[ ICMP ]###
     type      = time-exceeded
     code      = ttl-zero-during-transit
     chksum    = 0xbf6e
     unused    = 0
###[ IP in ICMP ]###
        version   = 4L
        ihl       = 5L
        tos       = 0x0
        len       = 52
        id        = 57161
        flags     = DF
        frag      = 0L
        ttl       = 2
        proto     = tcp
        chksum    = 0xcf32
        src       = 134.59.129.241
        dst       = 173.194.20.89
        \options   \
###[ TCP in ICMP ]###
           sport     = 43843
           dport     = http
           seq       = 3927922380L
           ack       = 3188073609L
           dataofs   = 8L
           reserved  = 0L
           flags     = A
           window    = 14165
           chksum    = 0x51f9
           urgptr    = 0
           options   = [('NOP', None), ('NOP', None), ('Timestamp', (5088093, 1579045454))]
###[ Padding ]###
              load      = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x9d\xeb\x00\x08\x01\x01\x00\nA\x01'





    ###[ IP ]###
  version   = 4L
  ihl       = 5L
  tos       = 0x0
  len       = 168
  id        = 37758
  flags     = 
  frag      = 0L
  ttl       = 246
  proto     = icmp
  chksum    = 0xaa73
  src       = 193.51.189.2
  dst       = 134.59.129.241
  \options   \
###[ ICMP ]###
     type      = time-exceeded
     code      = ttl-zero-during-transit
     chksum    = 0x2e1c
     unused    = 4
###[ IP in ICMP ]###
        version   = 4L
        ihl       = 5L
        tos       = 0x0
        len       = 60
        id        = 53079
        flags     = DF
        frag      = 0L
        ttl       = 5
        proto     = tcp
        chksum    = 0x6d73
        src       = 134.59.129.241
        dst       = 74.125.230.71
        \options   \
###[ TCP in ICMP ]###
           sport     = 45799
           dport     = http
           seq       = 2382327024L
           ack       = 0
           dataofs   = 10L
           reserved  = 0L
           flags     = S
           window    = 14600
           chksum    = 0x83ed
           urgptr    = 0
           options   = [('MSS', 1460), ('SAckOK', ''), ('Timestamp', (5088167, 0)), ('NOP', None), ('WScale', 4)]
###[ Padding ]###
              load      = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00X\xf6\x00\x08\x01\x01\x04\x01\x81\xff'

To clarify are you saying the you are pinging from R1 to R4 where the topology is R1->-R2->-R3->-R4 and R3 is generating the time-exceeded message, or R2 is generating the message? — jwbensley, Mar 19 '13 at 11:45
Also, would the last hop be on the same physical router, just another loopback interface nearer to the sender? — Alexander Janssen, Mar 19 '13 at 11:55
No no, I'm not talking about `echo-request` or `echo-reply` packets. Only `time-exceeded`. I found ICMP time exceeded packets whose payload contained an IP header with TTL=2 instead of TTL=1, which means that it wasn't when TTL=1 that the packet was dropped and an ICMP message was generated, but one hop before. Anyway, I don't have control over those routers, so I don't know if I'm going through multiple interfaces over the same physical router. Is there any way to check this? — Ricky Robinson, Mar 19 '13 at 13:02
Since you mention you are 'forging' low TTL values in your packets in another comment, can you provide the returned packets you are asking about? Or you can check http://www.nanog.org/meetings/nanog45/abstracts.php?pt=MTE4NSZuYW5vZzQ1 for how to interpret a traceroute. — cpt_fink, Mar 27 '13 at 01:12
Updated my answer... you're running into MPLS TTL expiration. I was able to capture a similar packet (TTL of 2 in a TTL-expired return). — cpt_fink, Mar 28 '13 at 03:33
@RickyRobinson You have two packets here, there first has TTL=2 and the second TTL=5, this is what you are referring to in this question yes? Not the outer TTL value which is 243 and 246 respectively, in both packets, yes? — jwbensley, Mar 28 '13 at 21:37
Yes, I'm referring to the TTL in the IP header contained in the ICMP payload, which in the two examples above is first 2 and then 5. — Ricky Robinson, Mar 29 '13 at 01:18

cpt_fink · Accepted Answer · 2013-03-29T03:46:09.247

6

http://packetlife.net/blog/2008/dec/22/disabling-mpls-ttl-propagation/

http://www.ciscopress.com/articles/article.asp?p=680824&seqNum=4

Your packets are MPLS-encapsulated when the outer label's TTL is decremented to 0, but the inner packet TTL is not updated, so the TTL-expired labeled packet is forwarded on and the internal IP packet (with an apparently valid TTL) is returned to you as expired by the final MPLS router.

============================

When a labeled packet TTL expires the packet is actually forwarded on until the end of the 'tunnel' it's on, since the router that decremented the TTL field to 0 might not have a valid route back to the original sender. So the MPLS label is edited to indicate TTL expiration and eventually the final tunnel router decapsulates the 'valid but label-expired' packet and sends it back with a TTL failure message.

Disclaimer: I read through TTL-relevant sections of several RFC's but nothing was definite on this handling, so I would say that this behavior could vary from vendor to vendor.

Evidence from a captured packet:

Internet Control Message Protocol
Type: 11 (Time-to-live exceeded)
Code: 0 (Time to live exceeded in transit)
Checksum: 0xf4df [correct]
Internet Protocol, Src: 192.168.1.x (192.168.1.x), Dst: 8.8.8.8 (8.8.8.8)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x80 (DSCP 0x20: Class Selector 4; ECN: 0x00)
    Total Length: 92
    Identification: 0x6b56 (27478)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 2  <===== payload of packet entering MPLS tunnel
    Protocol: ICMP (1)
    Header checksum: 0x7abb [correct]
    Source: 192.168.1.x (192.168.1.x)
    Destination: 8.8.8.8 (8.8.8.8)
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xf78f [correct]
    Identifier: 0x0001
    Sequence number: 111 (0x006f)
    Sequence number (LE): 28416 (0x6f00)
    Data (64 bytes)
MPLS Extensions
    Version: 2
    Reserved: 0x000
    Checksum: 0x5581 [correct]
    MPLS Stack Entry
        Length: 0x0008
        Class: 1
        C-Type: 1
        Label: 1864, Exp: 4, S: 1, TTL: 1
            0000 0000 0111 0100 1000 .... = Label: 1864
            .... .... .... .... .... 100. = Experimental: 4
            .... .... .... .... .... ...1 = Stack bit: Set
            Time to live: 1    <========== MPLS TTL

edited Mar 29 '13 at 03:46

answered Mar 23 '13 at 01:32

cpt_fink

907
5
12

THanks a lot! Why are you saying that my packets are "mpls-encapsulated" when the IP ttl is 0? If I got it right, when my data packet enters an MPLS tunnel, its IP TTL is copied onto the newly created label TTL. If this label TTL expires *inside* the MPLS tunnel, then an ICMP packet is generated. This ICMP message, as soon as it gets out of the MPLS tunnel, shows in its payload the TTL that my data packet had *before* entering the MPLS tunnel. Is that correct? – Ricky Robinson Mar 28 '13 at 15:27
Or are you saying that the ingress MPLS router doesn't drop an incoming packet with IP ttl=1, but it encapsulates it with a label and sends it through the MPLS tunnel? But in that case, if `# no mpls ip propagate-ttl` isn't active, that is in the default case, the IP TTL is copied onto the label TTL, so it expires anyway. When instead `# no mpls ip propagate-ttl` is on, *egress* router will see an IP packet with TTL=1 and generate an ICMP packet containing an IP header of ttl=1, not higher, – Ricky Robinson Mar 28 '13 at 15:35
@RickyRobinson At any point, when a packet travels down an MPLS tunnel and drops out because the TTL reaches 0, weather in the middle or at the end, the ICMP generating router will perform an IP lookup on the source and generate a packet with a TTL that is high enough to reach that source. So if the packet gets dropped at the end of the tunnel it won't sent it with a TTL=1, because it wouldn't make back to the source. – jwbensley Mar 28 '13 at 21:43
Yeah, of course. I was referring to the TTL *inside* the ICMP payload, that is to say the TTL of the packet that triggered the generation of that ICMP message. I'm not talking about the the TTL of the IP hearder that carries the whole ICMP message. – Ricky Robinson Mar 29 '13 at 01:23
Editing my answer... this is too long for a comment. – cpt_fink Mar 29 '13 at 03:28
Thanks a lot! So, please correct me if I'm wrong, what happens is following: for example, I send my IP packet `[IP: ttl=5]` and three hops away from here it reaches the ingress router of an MPLS tunnel. My `[IP: ttl=2]` packet is added a label by such router, and it becomes: `[label: ttl=2 [IP:ttl=2]]`. This packet enters the MPLS tunnel and the ttl is soon decremented to 0: `[label: ttl=0, ttl expired! [IP: ttl=2]]`. As soon as it reaches an egress router, its label is discarded, but because it contained a "ttl-expired" flag, an ICMP message has to be generated. – Ricky Robinson Mar 29 '13 at 09:50
So since the packet is `[IP: ttl=2]`, the corresponding ICMP message has to be: `[IP: ttl=255, proto=ICMP [ICMP type=11, code=0 [IP: ttl=2]]]`, which is exactly what I receive as an answer on my machine. Now, I guess it doesn't make much of a difference if the ICMP message goes back through the tunnel or not. I guess it does go back that way, but it has a proper and high enough TTL value, so it arrives at destination (me!) without any problems. – Ricky Robinson Mar 29 '13 at 09:56
Correction: *soon decremented to 1 (instead of to 0) – Ricky Robinson Mar 29 '13 at 09:59

score 4 · Answer 2 · answered Mar 20 '13 at 12:19

4

Routers are supposed to decrement the time to live field by 1 for each second spent processing the packet, but in no case should it decrement by less than 1.

So if a router spends more than a second processing a packet, it should decrement the TTL by more than one. However, it's exceedingly rare that a router would spend more than a second processing a packet, unless it was terribly bogged down.

Barring router implementation bugs, that's about the only thing I can think of that would explain this.

answered Mar 20 '13 at 12:19

Michael Hampton

244,070
43
506
972

No, I'm also checking the RTT for each (packet, corresponding ICMP message) pair that I have, and it's never greater than 0.01s, often much smaller. It looks like some routers, when they receive a packet with a low TTL value (actually as high as 5, I've noticed), don't let the packet through, but they drop it and they generate an ICMP time-exceeded packet. Otherwise, why would the payload of some packets contain an IP header with ttl=5? – Ricky Robinson Mar 20 '13 at 13:31
Have you tried running debug ip icmp on the router that is dropping packets and checking the logs as it is happening? – GerryEgan Mar 21 '13 at 19:28
I don't have control over that router. – Ricky Robinson Mar 22 '13 at 09:41
In any case, I was dealing with an MPLS router (the ICMP message I reported ends with 12 Bytes representing ICMP extension for MPLS) – Ricky Robinson Aug 14 '13 at 14:04

score 1 · Answer 3 · answered Mar 22 '13 at 20:28

1

While I can't say exactly what it is going on here, the Time Exceeded packet is sent on one of two conditions: TTL is exceeded, or fragment reassembly time is exceeded. What is the code sent back (second bite of the payload). If it is 1, the reason for sending the packet is reassembly time exceeded. This should normally be set at 60 to 120 seconds, not at 0.01s. Is it allways the same router(s) sending these packets back? Can you post the complete packet you are getting back and? Can you post any info on the routers in questions? Make? Model?

answered Mar 22 '13 at 20:28

JelmerS

777
6
12

The ICMP code is always 0, which means it's not a Reassembly Timeout. I don't have control over the router, so I don't know model or brand. It looks like this one router handles all the packets with a low ttl value and generates an ICMP packet on behalf of other routers, perhaps in the same network. – Ricky Robinson Mar 26 '13 at 18:58
My previous question remains: Can you post the entire packet you are getting back? – JelmerS Mar 27 '13 at 10:53
Yes, sorry. Now you can see two of those packets above. – Ricky Robinson Mar 27 '13 at 15:18

score 1 · Answer 4 · answered Mar 23 '13 at 07:13

While possible, a bug at this rudimentary level is unlikely ; plus considering the (fast) time it takes to process packets, I would rather direct the likeness of a problem towards some kind of loop or bouncing. In the trivial / usual case, the router decrements the TTL and directs the packet to the right interface based on its routing table.

There are some more complex cases where the TTL may depend upon the router implementation. From the top of my head, comes NAT masquerading where the packet may "reenter" the router after the address translation has been performed - this is typically the case when one wants to test a router WAN IP from the inside (and this doesn't work on all routers)

For instance,

source:
local  10.1.2.3/24 
destination:
public: 198.252.206.16/32 (an example..)

the address 198.252.206.16 being one's own address as seen from the Internet (the WAN side of the router) which is bound internally to 10.1.2.3 ie is actually the same host as the sender

The router receives the packet and realizes the WAN address is its own address, the packet "reenters" the WAN interface (implementation dependent) and is driven to the LAN address 10.1.2.3.

How is the TTL is treated in that case (which is not working on all routers) is implementation dependent.

score 1 · Answer 5 · answered Mar 26 '13 at 15:59

1

The answer to "Why are you receiving time-exceeded packets" (your original question of "How so? Any reason behind this?") is easy to answer:

Look at the time exceeded packet you caputured, what is the code value inside. If it is 0, it was a TTL issue at the generating router, if it was 1 it was a fragmentation issue at the generating router.

The question "How do you set this in a CISCO router?" doesn't make an sense, set what? No out of the ordinary behaviour has been displayed according to what you have said.

"Why is the router having TTL or fragmentation issues?" is a good question here I believe. If the router (assuming its always the same one) is out of your control then we can't say for sure. But we can speculate a little bit. It could be an MTU mismatch between interfaces, it could be a buffering problem I suppose also. This is assuming its a fragmentation issues. If it's a TTL issue, it could be a misconfiguration of an MPLS LSP or an MPLS LSR which is giving a conflicting ICMP reading due to PHP/UHP etc (although unlikely).

When you are receiving these time exceeded messages, are the current UDP/TCP flows having issues, do any of them drop? The time exceeded message should contain part of the data packet which caused the generation of the ICMP packet, is the original data unit a jumbo frame or large TCP packet, does it have the DF bit set?

You haven't given much to go on in terms of the context in which the ICMP packets are being generated in the first place.

answered Mar 26 '13 at 15:59

jwbensley

4,202
11
58
90

Thank you for your answer. The only thing is that I'm not dealing with ICMP type 11 code 1 packets. Sorry for not making it too clear in my initial post. These packets are always code 0, so no it's not a reassembly timeout problem. – Ricky Robinson Mar 26 '13 at 18:59
As for how I generate ICMP packets, I just replay a packet trace where I forged the TTL field with a low value. I'm trying to measure round trip times to each hop to destination. – Ricky Robinson Mar 26 '13 at 19:01
On a side note: the TTL value inside an ICMP payload is never 0, it's always 1. – Ricky Robinson Mar 27 '13 at 15:10
1

Given the above info in your comments I still maintain that no one can really tell you what's wrong here as you don't have access to the routers. It could be anything like MPLS (as I mentioned) or GRE tunnels causing issues, (un-)ECMP routing, or a problem with how the router's are handling the ICMP packets in their code (software bug), or they could be using a CoPP like system that is braking the ICMP flow. If you can't access or control them it will be very difficult to resolve this. I would like to know should you find an answer. I am going to think on this more, it is annoying me now :) – jwbensley Mar 27 '13 at 17:27
Thanks a lot! I'll look into those things you mentioned. What baffles me is that out of three locations in which I tested this, two showed this behaviour, so it can't be just an unfortunate router configuration. At least two! :) – Ricky Robinson Mar 27 '13 at 20:06

Generating ICMP packets when TTL=2?

5 Answers5