2

I'm trying to run this tutorial XDP code that is provided in the bcc.

The code I use is this script: bcc/examples/networking/xdp/xdp_drop_count.py.

and to my understanding, XDP flag works as follows (from that question):

#define XDP_FLAGS_SKB_MODE (1U << 1)
#define XDP_FLAGS_DRV_MODE (1U << 2)
#define XDP_FLAGS_HW_MODE (1U << 3)

So, doesn't this mean that if I change the flags bit to 

flags |= 1 << 3

I should be able to run this code in hardware accelerated mode (offloaded)?

I have a NIC card that supports XDP HW accelerated mode and it works fine when I just attach a simple program with only one line of code:

return XDP_PASS;

and attach it in offloaded mode by using ip link set dev interface xdpoffload etc.

So I have confirmed my NIC is capable of loading an offloaded XDP program but when I try the above, it gives me an error:

bpf: Attaching prog to enp4s0np1: Invalid argumentTraceback (most recent call last) :
File "xdp_drop_count.py", line 132, in <module>
b. attach_xdp(device, fn, flags)
File "usr/lib/python2.7/dist-packages/bcc/__init__.py", line 723, in attach_xdp % (dev, errstr))
Exception : Failed to attach BPF to device enp4s0np1: No such file or directory

Also, when I set the flags to :

flags |= 1 << 2

I am not sure if this is actually running the XDP program in driver mode.

Am I missing something?

Thank you in advance.

Qeole
  • 8,284
  • 1
  • 24
  • 52
Rosè
  • 345
  • 2
  • 13
  • 1
    The StackOverflow link you provided looks incorrect. – pchaigno Aug 07 '19 at 07:58
  • 1
    Probably [that link](https://stackoverflow.com/questions/57171394/with-attach-xdp-does-flags-control-the-mode). XDP program _is_ run in driver mode with your `1 << 2` flag, you can check that with e.g. `bpftool net`. – Qeole Aug 07 '19 at 08:48
  • 1
    @Qeole There seem to be some issue though, because it's *always* running in driver mode, even with `1 << 1` or `1 << 3`... – pchaigno Aug 07 '19 at 09:00
  • 1
    @pchaigno Uuuh not on my setup, the `-S` option does switch to skb mode for me. I'm hitting the same issue as OP for HW offload though. – Qeole Aug 07 '19 at 09:03
  • 2
    The program needs to be offloaded when it is loaded. The `XDP_FLAGS_HW_MODE` flag is not enough to offload the program - it's just used to tell to attach an offloaded program in XDP hardware mode. Because `b.load_func()` does not offload the program in the current case when it is loaded on the system, attaching it as XDP hardware offload fails when we call `b.attach_xdp()`. (Also per-CPU arrays are not supported for offload). Not sure if `b.load_func()` supports offload at all at this time, I don't think we've worked with bcc before. – Qeole Aug 07 '19 at 09:12
  • bcc does not explicitly support hardware offloading, so `load_func()` doesn't offload the program. We probably can add support though. – pchaigno Aug 07 '19 at 09:21
  • @pchaigno and Qoele Thank you so much for everything. I learned a lot by reading your comments! – Rosè Aug 07 '19 at 12:57
  • I've opened a pull request at https://github.com/iovisor/bcc/pull/2502. You should be able to offload BPF programs to your hardware card by pulling and building my branch locally. The pull request will likely evolved before it is merged in bcc. There are also some breaking API changes, so I'm not sure it will be accepted. – pchaigno Aug 29 '19 at 17:26
  • Merged two days ago, thanks @pchaigno! :) – Qeole Sep 05 '19 at 16:58

2 Answers2

2

For a BPF program to be attached as a XDP program, it needs to be offloaded to the NIC first, when being loaded on the system.

In your case, the b.load_func() provided by bcc does not support any option for offloading programs when passing them to the kernel. So when you later call b.attach_xdp() with the XDP_FLAGS_HW_MODE, the function fails, because it cannot find any program offloaded on the NIC.

Right now there is no workaround for offloading program with bcc. As pchaigno mentioned, the function simply does not offer an option to indicate the program should be offloaded.

It should not be too difficult to add support for offloading programs to bcc though, so it should probably be available in the future (especially if pchaigno feels like adding it :p). You would still need to replace the per-CPU array by a regular array in your program, as the former is not supported for offload at this time.

Regarding the mode in which your programs run, this is something you can check with bpftool net for example.

Qeole
  • 8,284
  • 1
  • 24
  • 52
  • Thank you so much for helping me out! Then is there no way for a HW offloaded XDP program to communicate with the user space? Because to my understanding, you need to use BCC to use eBPF map and have XDP kernel space and user space programs communicate. I am terribly sorry if my understanding is wrong :( – Rosè Aug 07 '19 at 12:56
  • 1
    You don't need bcc at all. Bcc is only a framework to make it easier to work with BPF programs (and mostly oriented towards tracing programs). All the interaction is done through the `bpf()` syscall, but you can use libbpf (if you write a C app) to wrap around it, or from command line, bpftool to interact with programs and maps. You just won't have the same syntactic sugar (`.lookup()`, `b.get_table()` etc.) without bcc, that's all. Have a look at e.g. [our BPF samples](https://github.com/Netronome/bpf-samples/): L4LB uses bpftool, RSS uses libbpf to communicate with (possibly offloaded) maps. – Qeole Aug 07 '19 at 13:07
  • Thanks a lot for your help! I really appreciate it! – Rosè Aug 16 '19 at 13:55
2

If you build bcc from sources

Since commit d147588, bcc has hardware offload support. To offload programs using bcc, you will need three things:

  • The XDP_FLAGS_HW_MODE bit (1U << 3) should be set in the flags passed to attach_xdp().
  • The name of the interface to which you want to offload the program should be given to BPF() with the device= parameter. It will allow bcc to offload the maps to the appropriate device. It is unnecessary if you don't have maps.
  • The interface's name should also be given to load_func, again with parameter device=, such that bcc tells the kernel where to offload the program.

Note that, with the latest bcc sources, the xdp_drop_count.py script has been updated to do all this for you when you pass the -H option:

sudo ./xdp_drop_count.py -H $ETHNAME

For older versions of bcc

Older versions of bcc do not support hardware offload. You can use bpftool or ip (>v4.16) instead, e.g.:

sudo ip link set dev $ETHNAME xdpoffload obj prog.o sec .text
sudo bpftool prog load prog.o /sys/fs/bpf/prog type xdp dev $ETHNAME
pchaigno
  • 11,313
  • 2
  • 29
  • 54