XFRM Ipsec Anti-Replay detection for ESN packets

Question

For IPSEC anti-replay detection, if the sequence number is less than the lowest sequence in the window, is the packet dropped or accepted?

I assume it should be dropped, but for ESN packets it seems that the packet is accepted. In the xfrm implementation in the linux kernel if seq < bottom then the packet is accepted.

Below I have put the code for reference: https://elixir.bootlin.com/linux/v4.18-rc3/source/net/xfrm/xfrm_replay.c#L435

static int xfrm_replay_check_esn(struct xfrm_state *x,
             struct sk_buff *skb, __be32 net_seq)
{
    unsigned int bitnr, nr;
    u32 diff;
    struct xfrm_replay_state_esn *replay_esn = x->replay_esn;
    u32 pos;
    u32 seq = ntohl(net_seq);
    u32 wsize = replay_esn->replay_window;
    u32 top = replay_esn->seq;
    u32 bottom = top - wsize + 1;

    if (!wsize)
        return 0;
    if (unlikely(seq == 0 && replay_esn->seq_hi == 0 &&
         (replay_esn->seq < replay_esn->replay_window - 1)))
        goto err;

    diff = top - seq;

    if (likely(top >= wsize - 1)) {
        /* A. same subspace */
        if (likely(seq > top) || seq < bottom)
            return 0;
    } else {
        /* B. window spans two subspaces */
        if (likely(seq > top && seq < bottom))
            return 0;
        if (seq >= bottom)
            diff = ~seq + top + 1;
    }

I am not sure what RFC4303 is expecting too:

Under Case A: If Seql >= Bl (where Bl = Tl - W + 1) AND Seql <=
Tl, then check the corresponding bit in the window to see if
this Seql has already been seen.  If yes, reject the packet.  If
no, perform integrity check (see Appendix A2.2. below for
determination of Seqh).

If the sequence number if below the window it should be dropped. If Linux's XFRM isn't doing that then this should be a security report. — Thomas M. DuBuisson, Jul 13 '18 at 13:38
@ThomasM.DuBuisson No, that's not the case. The code only compares the lower 32 bits of the 64 bit ESN. I've tried to clarify this in my answer. — ecdsa, Jul 13 '18 at 15:32

score 4 · Answer 1 · edited Oct 07 '21 at 08:57

If the lower 32 bits of the sequence number (the part that's transmitted in the ESP header) are lower than bottom it's assumed that the sequence number subspace is moved (i.e. the upper 32 bits have increased). That's covered in section A2.2. of RFC 4303:

Under Case A (Figure 1):
    If Seql >= Bl (where Bl = Tl - W + 1), then Seqh = Th
    If Seql <  Bl (where Bl = Tl - W + 1), then Seqh = Th + 1

There is pseudocode that shows handling this in section A2.3. (the Else block of the If (Seql >= Tl - W + 1) statement in the If (Tl >= W - 1) block). In the kernel code you see something similar in xfrm_replay_seqhi.

After the sequence number check the packet's integrity is verified using the complete 64 bit sequence number (with the upper 32 bits increased by one if the received sequence number was below the window). If this fails, the packet is dropped, otherwise the window is moved (xfrm_replay_advance_esn) and might now span two subspaces (Case B in the RFC).

XFRM Ipsec Anti-Replay detection for ESN packets

1 Answers1