Unable to unload BPF program

Question

I am unable to unload a BPF program from code. I am using the Cilium eBPF library to load the program and netlink to add the BPF function to an interface. Here's what I'm doing:

type BpfObjects struct {
    CollectIpsProg *ebpf.Program        `ebpf:"collect_ips_prog"`
}

    var objects BpfObjects

    // Load the BPF program
    spec, err := ebpf.LoadCollectionSpec("collect_ips.o")

    if err != nil {
        log.Fatalln("ebpf.LoadCollectionSpec", err)
    }

    if err := spec.LoadAndAssign(objects, nil); err != nil {
        log.Fatalln("ebpf.LoadAndAssign", err)
    }

    // Load to XDP
    link, err := netlink.LinkByName("enp0s8")

    if err != nil {
        log.Fatalln("netlink.LinkByName", err)
    }

    err = netlink.LinkSetXdpFdWithFlags(link, objects.CollectIpsProg.FD(), 2)

    if err != nil {
        log.Fatalln("netlink.LinkSetXdpFdWithFlags:", err)
    }
    ...

    // Cleanup. This does not unload the BPF program
    objects.CollectIpsProg.Close()
    objects.CollectIpsProg.Unpin()

Even though I am closing the program, bpftool prog and xdp-loader status still show the BPF program. I can unload the program using bpftool or xdp-loader.

score 3 · Accepted Answer · answered Feb 09 '22 at 08:31

3

eBPF programs only unload when there are no more references to it(File descriptors, pins), but network links also hold their own references. So to unload the program, you first have to detach it from your network link.

You can do so by setting the program fd to -1:

err = netlink.LinkSetXdpFd(link, -1)
if err != nil {
    log.Fatalln("netlink.LinkSetXdpFd:", err)
}

answered Feb 09 '22 at 08:31

Dylan Reimerink

5,874
2
15
21

Thanks, I new I had to do something with netlink but couldn't figure out what. They should document this in the function. – user2233706 Feb 09 '22 at 14:26
This worked on my VM, but the program is not removed when I run this on our server. ```bpftool prog``` still shows the program. I can't even remove it if I use ```bpftool net detach xdp dev```. The server is running kernel 5.16.7-1. – user2233706 Feb 09 '22 at 15:45
Are you using a prog_array by any chance? These can also hold references to programs causing them to stay loaded. Just check everything that can hold references(pins in the bpffs, netlinks `ip link show`, programs and maps). There is no surefire way to know how holds what, the kernel just has an internal ref counter which increments on new references and decrements when they are gone/closed, it unloads the program if the ref count reaches 0 – Dylan Reimerink Feb 09 '22 at 15:53
Yes, I'm using a prog_array. I don't explicitly close the map, but it is closed when I exit the program on the VM, but not the server. I explicitly closed the prog_array, but it makes no difference on the server. Can you unpin a map using bpftool? – user2233706 Feb 09 '22 at 16:59
You should be able to just `rm` the pin since it is a pseudo-file. Also you can remove the reference from the prog_array by deleting the element which contains it, just so you know. – Dylan Reimerink Feb 09 '22 at 17:35
My prog array is empty after the program exits. I don't see any other programs loaded other than the program I'm trying to remove. I'll create a new question on this. – user2233706 Feb 09 '22 at 18:29

Unable to unload BPF program

1 Answers1