0

In my XDP program I observe that bpf_map_update_elem() does not update all CPUs for a BPF_MAP_TYPE_LRU_PERCPU_HASH. Here's my map:

struct {
    __uint(type, BPF_MAP_TYPE_LRU_PERCPU_HASH);
    __uint(key_size, sizeof(__u32));
    __uint(value_size, sizeof(__u64));
    __uint(max_entries, 1<<22);
} ips_state SEC(".maps");

Here's how I insert into the map:

__u64 value = bpf_ktime_get_ns();
long status = bpf_map_update_elem(&ips_state, &iph->saddr, &value, BPF_ANY);

When I dump the table using bpftool, I see that only one of the CPUs have a value:

key:
0a 02 01 66
value (CPU 00): 00 00 00 00 00 00 00 00
value (CPU 01): 00 00 00 00 00 00 00 00
...
value (CPU 24): 4f f1 14 a6 1e 6a 5c 00
value (CPU 25): 00 00 00 00 00 00 00 00
...

When I attempt to read from the map in the XDP program, the program reads from CPU other than 24, and I get a value of 0.

How can I update all the CPUs in XDP? I don't see an API specific to updating all the CPUs. Or does this have to be done in user space?

user2233706
  • 6,148
  • 5
  • 44
  • 86

1 Answers1

2

The behavior you are describing is actually a feature of the PERCPU variant of the map. Giving each CPU its own piece of memory allows all CPUs to change values in the map without the need for locking or atomics which increases performance for write-heavy use cases.

If you want to share data across multiple CPUs you will need to use the non-PERCPU variant and use spinlocks or atomics to avoid race conditions.

Lastly, in v5.19 the bpf_map_lookup_percpu_elem helper was added to allow programs to read the map elements of PERCPU maps for other CPUs.

Dylan Reimerink
  • 5,874
  • 2
  • 15
  • 21
  • In [this answer](https://stackoverflow.com/a/72192140/2233706) you mention that map modifying operations are atomic. Can you clarify whether you need a spinlock when modifying a non-PERCPU map? – user2233706 Mar 16 '23 at 15:44
  • 1
    Using the update helper is basically a RCU operation, you read the current value, but are not allowed to update it directly, so you would have to copy it into the stack of the program + modification and then call the update helper with this copy. That could be very expensive computation wise vs spinlocking which allows you to modify the map values directly without doing a copy. All of this assumes you cannot use atomics to achieve the same. If you only have to increment or replace a single value then you can use atomics, but for multi field modification RCU or locking is required – Dylan Reimerink Mar 17 '23 at 09:00