SO_ATTACH_REUSEPORT_CBPF socket option unexpected behavior

Question

I am trying use a port from two applications and have each of them receive the packet from a different set of IP addresses. In order to achieve this, I use the SO_REUSEPORT and SO_ATTACH_REUSEPORT_CBPF socket options. My code is as follows:

parentfd = socket(AF_INET, SOCK_STREAM, 0);
if (parentfd < 0)
  error( "ERROR opening socket");

struct sock_filter code[]={
  { 0x28, 0, 0, 0x0000000c },
  { 0x15, 0, 3, 0x00000800 },
  { 0x20, 0, 0, 0x0000001a },
  { 0x15, 2, 0, 0xc0a8ff01 },
  { 0x6, 0, 0, 0x00000000 },
  { 0x6, 0, 0, 0x00040000 },
  { 0x6, 0, 0, 0x00000001 },
};

struct sock_fprog bpf = {
  .len = ARRAY_SIZE(code),
  .filter = code,
};

if (setsockopt(parentfd, SOL_SOCKET, SO_REUSEPORT, (const void *)&optval,sizeof(optval)))
  error("ERROR setting SO_REUSEPORT");

if (setsockopt(parentfd, SOL_SOCKET, SO_ATTACH_REUSEPORT_CBPF, (const void *)&bpf, sizeof(bpf)))
  error("ERROR setting SO_ATTACH_REUSEPORT_CBPF);

I also have a different process that listens to the same port using only the SO_REUSEPORT flag. From a machine with IP 192.168.255.1 I am running echo 1234 | ncat 192.168.255.150 1234. Based on my filter I would expect all traffic from that IP address to be received by the second process. However, it is all received by the first. When I change the filter to a simple:

struct sock_filter code[]={ { 0x6, 0, 0, 0x00000001 }, };

It works as expected and all packets are received by the second process. Any idea why this might be happening?

How you came up with the original filter? – Tarun Lalwani Nov 27 '17 at 13:29 — Tarun Lalwani, Nov 27 '17 at 13:29

score 3 · Accepted Answer · answered Dec 04 '17 at 22:07

3

I found out what the problem was. The filter is applied to all packets, even the TCP handshake packets. Also, the base pointer points to the first byte of the packet payload, not headers. Hence, when it executes

ldh[12]

it gets out of the limits of the packet (SYN packet has 0 bytes of payload) and the default behavior is to return 0.

answered Dec 04 '17 at 22:07

user2424276

591
1
5
24

You need to use a special negative offset to access the headers (SKF_NET_OFF) and then add from there. So, what you probably want to do is check the protocol first (SKF_AD_OFF+SKF_AD_PROTOCOL) and then use SKF_NET_OFF+12. – Compholio Jul 27 '21 at 16:49

score 0 · Answer 2 · answered Dec 03 '17 at 20:01

The non-working code is:

l0: ldh [12]                   /* read EtherType (2 bytes), which is found at offset 12 (decimal) */
l1: jeq #0x800, l2, l5         /* if EtherType == `0x800` (IPv4), jump to `l2`, otherwise jump to `l5` */
l2: ld [26]                    /* read source IP address (4 bytes) */
l3: jeq #0xc0a8ff01, l6, l4    /* if source IP address == 192.168.255.1, jump to l6 (return 1), else jump to l4 (return 0) */
l4: ret #0
l5: ret #0x40000
l6: ret #0x1

The working code is:

ret #0x1

socket (7) says:

The BPF program must return an index between 0 and N-1 representing the socket which should receive the packet (where N is the number of sockets in the group). If the BPF program returns an invalid index, socket selection will fall back to the plain SO_REUSEPORT mechanism.

On my machine tcpdump -i lo -ddd 'src host 192.168.255.1' produces

10
40 0 0 12
21 0 2 2048
32 0 0 26
21 4 5 3232300801
21 1 0 2054
21 0 3 32821
32 0 0 28
21 0 1 3232300801
6 0 0 262144
6 0 0 0

Which is

l0: ldh [12]
l1: jeq #0x800, l2, l4
l2: ld [26]
l3: jeq #0xc0a8ff01, l8, l9
l4: jeq #0x806, l6, l5
l5: jeq #0x8035, l6, l9
l6: ld [28]
l7: jeq #0xc0a8ff01, l8, l9
l8: ret #0x40000
l9: ret #0

I fail to see anything that's obviously wrong with your code.

Have you tried running tcpdump on the server? Perhaps you've forgotten to remove an extra IP address on the client, or there's a forgotten SNAT rule somewhere?

Which kernel version are you running? Could you post a minimal C application that reproduces the problem?

SO_ATTACH_REUSEPORT_CBPF socket option unexpected behavior

2 Answers2