1

When I try to attach a BPF program in XDP offload mode, I get Invalid argument. I get the same error if attach through code or by using bpftool. Here's how I'm attaching using netlink:

err = netlink.LinkSetXdpFdWithFlags(link, objects.CollectIpsProg.FD(), 8)

And from using bpftool:

# bpftool prog loadall collect_ips.o /sys/fs/bpf/collect_ips type xdp
# bpftool net attach xdpoffload id 106 dev public
Error: interface xdpoffload attach failed: Invalid argument

I don't have any issues loading the program in driver mode, where 4 is passed to LinkSetXdpFdWithFlags.

My NIC, Mellanox MT28800 Family [ConnectX-5 Ex], should support HW offload.

My main XDP program makes calls to two different tail programs. I use BPF_MAP_TYPE_RINGBUF, BPF_MAP_TYPE_PROG_ARRAY, and BPF_MAP_TYPE_ARRAY.

user2233706
  • 6,148
  • 5
  • 44
  • 86

1 Answers1

1

Mellanox cards support some hardware offload (e.g., flow control rules), but not the offload of BPF programs as far as I know. The only Ethernet adapters out there that support BPF offloading are Netronome's cards.

One way to check this is to grep for the XDP_SETUP_PROG_HW BPF netdev command in the Linux source code:

$ git grep XDP_SETUP_PROG_HW
drivers/net/ethernet/netronome/nfp/nfp_net_common.c:    case XDP_SETUP_PROG_HW:
drivers/net/netdevsim/bpf.c:    if (bpf->command == XDP_SETUP_PROG_HW && !ns->bpf_xdpoffload_accept) {
drivers/net/netdevsim/bpf.c:    if (bpf->command == XDP_SETUP_PROG_HW) {
drivers/net/netdevsim/bpf.c:    case XDP_SETUP_PROG_HW:
include/linux/netdevice.h:      XDP_SETUP_PROG_HW,
net/core/dev.c: xdp.command = mode == XDP_MODE_HW ? XDP_SETUP_PROG_HW : XDP_SETUP_PROG;

That command is used to tell the driver to offload the BPF program to the hardware, via the ndo_bpf callback function.

pchaigno
  • 11,313
  • 2
  • 29
  • 54
  • I thought it was one of these [values](https://elixir.bootlin.com/linux/v5.16.7/source/include/uapi/linux/if_link.h#L1180). This means that netlink doesn't support HW offload? – user2233706 Feb 10 '22 at 19:09
  • Assuming 8 is the value for HW offload, I don't see any code in ```LinkSetXdpFdWithFlags``` or the functions it calls that would prevent it from using 8. – user2233706 Feb 10 '22 at 19:13
  • Oh, of course, you're right. And the problem is elsewhere; I had missed the network card you're using. I updated my answer. – pchaigno Feb 10 '22 at 20:58
  • Also added a way to check that in the source in case it ever changes. Note the netdevsim here is a driver for testing purposes only. – pchaigno Feb 10 '22 at 21:10
  • Is it the case the HW doesn't support offload or just the driver? – user2233706 Feb 10 '22 at 21:22
  • Probably the hardware, but maybe Mellanox has some NPU in some of their cards that could execute BPF programs with a bit of work. Note that in the case of Netronome a special JIT compiler is also needed to convert the BPF bytecode in a format that the card understands (including 32-bit registers). – pchaigno Feb 10 '22 at 21:27
  • Ok. What are the limitations of offloaded programs? For example, would user space get notifications on a ring buffer write. – user2233706 Feb 10 '22 at 21:37
  • 3
    From what I have found online, offloading is sort of limited, the full explanation doesn't fit in a comment. Here are some good links: - https://www.netronome.com/documents/305/eBPF-Getting_Started_Guide.pdf - https://www.youtube.com/watch?v=6WXsTixR4ak - https://archive.fosdem.org/2018/schedule/event/xdp/attachments/slides/2220/export/events/attachments/xdp/slides/2220/fosdem18_SdN_NFV_qmonnet_XDPoffload.pdf - https://www.netronome.com/documents/141/viljoen-xdpoffload-talk.pdf – Dylan Reimerink Feb 11 '22 at 09:05
  • Do you know if tail calls and support for all maps is still "future work?" Those documents are old. I assume driver is the most commonly used mode. – user2233706 Feb 11 '22 at 20:30
  • I haven't seen work from Netronome in the kernel in a long while, so they may indeed still be "future work". – pchaigno Feb 12 '22 at 13:27