2

So, as per title I'm trying to load an XDP program, when surprisingly the bpf verifier kicks in spitting in my face with the famous back-edge error:

libbpf: load bpf program failed: Invalid argument
libbpf: -- BEGIN DUMP LOG ---
libbpf: 
back-edge from insn 271 to 69

libbpf: -- END LOG --
libbpf: failed to load program 'xdp_prog'

Even though the only for-loop - with the number of iterations known at compile time - in my restricted ebpf C code is guarded by pragma unroll. Here's a code snippet showing the affected for-loop as defined inside an __always_inlined function:

#pragma unroll
for (i = 0; i < 8; i++)
{
    int k = idx + i;
    mask = bpf_map_lookup_elem(&a_map, &k);
    if (!mask || (mask->an_idx == 0))
        return -1;

    *m_key = *key;
    foo(m_key, mask);  // an __alwais_inline func

    id = bpf_map_lookup_elem(&b_map, m_key);
    if (id)
    {
        *out_id = *id;
        return 0;
    }
}

Maybe the problem is with clang failing to unroll the loop? If that's right, why does it fail, is there any workaround? It's unacceptable to manually unroll the loop as it results in a horrendous, unmaintainable, and unreadable code.

Oh, I'm working with:

  • kernel 4.19.3
  • llvm-clang 8

Any thoughts?

UPDATE
Just noticed that even the following dummy for-loop seems not to be unrolled, with the bpf verifier complaining about back-edge:

#pragma unroll
for (i = 0; i < 8; i++)
{
    int k = i;
    mask = bpf_map_lookup_elem(&a_map, &k);
}

Is it just me for this doesn't make any sense?

Kirill Lykov
  • 1,293
  • 2
  • 22
  • 39
pa5h1nh0
  • 262
  • 3
  • 13
  • 1
    Why “unsurprisingly” if you use the pragma? Why “clearly” if you're not 100% sure :) Some remarks: 1) Did you get a warning at compile time? I believe clang prints one when it cannot unroll a loop. 2) Did you have a look at the generated BPF instructions (`llvm-objdump -S objfile.o`) to make sure the back-edge comes from that specific loop? 3) If you are flexible on kernel versions, you may be interested to know that linux 5.3 (currently in development) has support for bounded loops. – Qeole Jul 03 '19 at 15:08
  • Agree with "unsurprisingly" to "surprisingly", also agree with "clearly" to "maybe". I'd edit that as a first thing. Answering your questions: 1) no warning at compile time; 2) pretty sure as it's the only "loop" defined in my code, also because the verification passes if commenting out this for-loop; 3) would that solve the problem? – pa5h1nh0 Jul 03 '19 at 15:42
  • Look in bpf_map_lookup_elem for EINVAL. – S.S. Anne Jul 03 '19 at 16:29
  • 3) provides support for this kind of loops in the BPF verifier, so a back-edge for this loop would not be rejected anymore. So yeah it should solve the problem. Sounds a bit like overkill though. In my experience I always got warnings ([like this one](https://stackoverflow.com/questions/55908053/why-does-clang-is-unable-to-unroll-a-loop-that-gcc-unrolls)) when clang failed to unroll :/. Can't really help more without the C or BPF code, sorry. (Also with Linux 4.19 you should not have to inline all functions, BPF functions calls are supported. But that's not related to the issue.) – Qeole Jul 03 '19 at 16:42
  • @JL2210 what does that even mean in this case? – pa5h1nh0 Jul 04 '19 at 09:34
  • @pa5h1nh0 Look at the source code for `bpf_map_lookup_elem` in the kernel source tree, and search for EINVAL. – S.S. Anne Jul 04 '19 at 14:57
  • I don't think the `EINVAL` is returned by `bpf_map_lookup_elem`, it wouldn't be related to a back edge? @pa5h1nh0 I often see `#pragma clang loop unroll(full)` instead of `#pragma unroll`, I thought there was no difference but maybe try the other one just in case? – Qeole Jul 08 '19 at 08:40
  • @Qeole yep, indeed ```bpf_map_lookup_elem``` returns a pointer, that's why "_looking for EINVAL_" makes no sense to me. Already tried ```#pragma clang loop unroll(full)``` version, same outcome, in fact clang documentation states that these two are synonyms. Atm I'm manually unrolling the loop, which is disgusting, oh well. – pa5h1nh0 Jul 08 '19 at 08:57
  • Would you have a standalone reproducer available somewhere, ideally with a Makefile to be sure we run the same commands, so I could try to reproduce on my side? – Qeole Jul 10 '19 at 08:28
  • By the way your “dummy-loop” example unrolls just fine for me :/. – Qeole Jul 11 '19 at 14:05
  • 1
    Update -- for the kernel 5.3.0+ unroll is not necessary https://lwn.net/Articles/794934/ – Kirill Lykov Jul 13 '22 at 12:11

1 Answers1

-2

I write the following code, and it works.

#pragma clang loop unroll(full)
for (int i = 0; i < 128; i++)
{
    if (bpf_map_lookup_elem(&conntrack_hash_tab, &reply_key) == NULL)
    {
        break;
    }
    reply_key.dport = reply_key.dport+1;
}
keniee van
  • 483
  • 4
  • 5