2

When a XDP program is loaded in SKB mode, I am unable to remove any maps and some programs from code. I can't remove these same BPF entities using bpftool either. Here's my code for cleaning up the entities:

type BpfObjects struct {
    CollectIpsProg *ebpf.Program        `ebpf:"collect_ips_prog"`
    HttpProg *ebpf.Program              `ebpf:"http_prog"`
    TlsProg *ebpf.Program               `ebpf:"tls_prog"`
    FlowRingBuffer *ebpf.Map            `ebpf:"flow_ring_buf"`
    JumpTable *ebpf.Map                 `ebpf:"jump_table"`
}

var objects BpfObjects
...

// None of these calls return an error

netlink.LinkSetXdpFdWithFlags(link, objects.CollectIpsProg.FD(), 2)

...

netlink.LinkSetXdpFd(link, -1)

objects.FlowRingBuffer.Unpin()
objects.JumpTable.Unpin()
objects.TlsProg.Unpin()
objects.HttpProg.Unpin()
objects.CollectIpsProg.Unpin()

objects.FlowRingBuffer.Close()
objects.JumpTable.Close()
objects.TlsProg.Close()
objects.HttpProg.Close()
objects.CollectIpsProg.Close()

On my VM that I use for development, I only need call do netlink.LinkSetXdpFd(link, -1) to clean up everything. However, when I run on my target server, I observe that program collect_ips_prog is still loaded and none of the maps are removed:

# bpftool prog
117: xdp  name collect_ips_pro  tag 6f70925233787bae  gpl
    loaded_at 2022-02-09T19:40:06+0000  uid 0
    xlated 1112B  jited 785B  memlock 4096B  map_ids 148,149,150
    btf_id 90

# bpftool map
148: percpu_array  name bpf_context_tab  flags 0x0
    key 4B  value 276B  max_entries 1  memlock 24576B
149: prog_array  name jump_table  flags 0x0
    key 4B  value 4B  max_entries 2  memlock 4096B
    owner_prog_type xdp  owner jited
150: ringbuf  name flow_ring_buf  flags 0x0
    key 0B  value 0B  max_entries 16777216  memlock 0B

collect_ips_prog is the only program I load to the interface using netlink.LinkSetXdpFdWithFlags. I populate the other two programs in jump_table in code. After the program exits, the jump_table map is empty:

# bpftool map dump name jump_table
Found 0 elements

The program is still there if I do bpftool net detach xdp dev eth0. There is no error/error code.

I do not have this issue when I load the XDP program in driver mode.

The server is running CentOS 7, and I replaced the kernel with 5.16.7-1.el7.elrepo.x86_64. My NIC is a Mellanox MT28800 Family [ConnectX-5 Ex].

user2233706
  • 6,148
  • 5
  • 44
  • 86
  • What is the model of the ethernet (e.g. are you using a netronome ethernet?) Also, is it a distro kernel that you are using? can you also provide the output of `xdp-loader unload eth0 -a -vv` – Ahmed Masud Feb 09 '22 at 20:51
  • I can't get ```xdp-loader``` to work on this box. My distro doesn't have libbpf and it doesn't support a newer version of clang. – user2233706 Feb 09 '22 at 21:17
  • 1
    Did you try to detach the XDP program with `ip`? It should be something like `ip link set dev eth0 xdp off`. – pchaigno Feb 10 '22 at 09:23
  • 2
    Your program being loaded in the kernel (i.e. showing up in `bpftool prog`) does not mean that it is still attached to your interface. It could be pinned in the bpffs for example (`ls /sys/fs/bpf/` as root to check), this would prevent it from being unloaded even if it's not attached and running. What does `bpftool net show` (or `ip link show dev eth0`) say? If it's still attached, could you try `bpftool net detach xdpgeneric dev eth0`? If it's still attached after that, can you please look at the output from `bpftool link`? – Qeole Feb 10 '22 at 11:03
  • @pchaigno That didn't work. – user2233706 Feb 10 '22 at 14:27
  • @Qeole ```bpftool net detach xdpgeneric dev eth0``` works. But how to do from code? – user2233706 Feb 10 '22 at 14:28
  • 1
    In `netlink.LinkSetXdpFdWithFlags(link, objects.CollectIpsProg.FD(), 2)`, do you know what the `2` stands for? I wonder if it may be a flag that stands for [the mode](https://elixir.bootlin.com/linux/v5.16/source/tools/bpf/bpftool/net.c#L63). If that's the case then you may want to try to reuse the same mode when detaching, in your example above `netlink.LinkSetXdpFdWithFlags(link, -1, 2)`? – Qeole Feb 10 '22 at 15:30
  • Thanks, that works. Feel free to post as answer. ```2``` comes from [here](https://elixir.bootlin.com/linux/v5.16.7/source/include/uapi/linux/if_link.h#L1181). These should be the flag values because if I passed ```4```, I would see a reference to ```driver``` in ```bpftool net```, versus ```generic``` if I passed ```2```. – user2233706 Feb 10 '22 at 15:50

0 Answers0